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Executive Summary 



The New York City Department of Youth and Community Development 
(DYCD) launehed serviees under its Out-of-Sehool Time (OST) Programs for 
Youth in September 2005, with the award of funds to support more than 500 
programs aeross New York City. Together, DYCD and the eity’s nonprofit 
eommunity, working elosely with the New York City Department of Edueation 
(DOE), extended serviees to more than 51,000 youth aeross the eity in the 
initiative’s first year, making OST almost certainly the nation’s largest after- 
school program effort. The OST initiative is designed to address a broad range of 
developmental objectives for youth and to serve the needs of New York City’s 
families and communities. 

This report of findings from the OST evaluation presents data on the first 
year of program implementation and early findings on participant engagement in 
OST programs and associated academic and social development outcomes. It 
describes a program effort that builds on DYCD’s historic commitment to New 
York City’s youth and on the work of nearly 200 community-based and other 
nonprofit organizations with lead roles in implementing the initiative. These 
findings are based on data collected from multiple sources, including the 
initiative’s participation-tracking database (OST Online), surveys of participants, 
program directors, program staff, parents, and executive directors of provider 
organizations, and visits to a sample of 15 OST programs. The evaluation 
addresses the following core questions: 

1 . What are the characteristics of programs supported by the OST 
initiative? 

2. Who participates in these programs, and what are their patterns of 
attendance? 

3. What are participants’ patterns of social and emotional growth? 

Do programs affect participants’ educational performance and, if 
so, how? 

4. Do programs meet the city’s needs for assistance to working 
parents and for improvement in community-level capacities to 
serve youth during the out-of-school hours and, if so, how? 

The OST evaluation is currently designed to collect and report on data 
spanning the initiative’s first three years of operations. The evaluation’s design 
recognizes that the initiative’s first year was focused on launching a very large 
number of programs throughout the city. Evaluators looked for program 
providers in Year 1 to hire needed staff, recruit and retain an initial cohort of 
youth participants, provide safe environments, and deliver activities and services 
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that generally responded to student and parent interests. As reported here, OST 
programs achieved these objectives. 



In the second and third years, evaluators’ expectations will be somewhat 
higher. In particular. Year 2 data on OST programs should provide evidence that 
providers are achieving at least some of the quality objectives articulated in 
DYCD’s 2004 Request for Proposals (RFP). Year 3 should generate preliminary 
evidence of positive early outcomes for participating youth. 

This executive summary focuses on findings from programs funded 
through Option I of the OST initiative. The full report also describes program 
implementation and participant engagement in Option II and Option III OST 
programs. 



Scope and Extent of OST Programming 

By the end of the 2005-06 school year, according to OST Online data, 528 
programs were launched and producing data on youth participation, with 417 sites 
operating Option I programs, 100 operating Option II programs, and 1 1 operating 
Option III programs. Among all sites, 314 were based in New York City schools, 
and 214 were based in other locations (and are known as center-based programs). 
DYCD awarded over $44 million to these 528 programs, with an average award 
of $84,000 per program and a median award of $73,000. Award amounts ranged 
from $3,000 to $340,000. Despite some delays in program start-up, the majority 
of these programs operated for at least nine months during the school year. 

Across the board. Option I programs were successful in enrolling 
participants during the first year of the OST initiative. As described in programs’ 
entries in OST Online, programs served 50,978 students from September 2005 
through June 2006. Across all grade levels. Option I programs served more 
students than were specified in their contracts, with programs funded to serve 
3 1,668 students and actually serving 40,818 students. Seventy-four percent of 
Option I programs met or exceeded their enrollment targets. 



Demographic Characteristics of Participants 

The majority of participants enrolled in all OST programs were in the 
elementary grades (44 percent) or middle grades (32 percent), according to OST 
Online. Twenty-four percent of participants were in grades 9-12. OST programs 
served approximately equal numbers of boys and girls. Across all OST options, 
Hispanic/Latino participants were the largest group served (39 percent of 
participants in all OST programs). 
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structural and Institutional Features of OST Programs 



In the first year of implementation of the OST initiative, programs worked 
to ereate program features that would allow them to deliver high-quality serviees 
to partieipants. These features included adoption of program policies and 
procedures, hiring of staff and development of staff capacity, allocation of 
program resources to maximize effectiveness, and establishment of strong 
partnerships with schools and other community institutions. Evidence of success 
in these areas met the evaluators’ expectations for comprehensiveness in first-year 
start-up efforts. 

Option I program directors almost unanimously reported in survey 
responses that providing a safe environment for youth was a major objective of 
their OST program (98 percent). They also overwhelmingly reported that they 
offered open enrollment to all youth who were interested in attending the program 
(91 percent). In addition, more than half (59 percent) of directors reported that 
they recruited youth who were recommended by school-day teachers. 

Program director and staff qualifications. OST program directors are 
typically responsible for the day-to-day management of the program, including 
supervising staff, developing program activities, recruiting participants, and 
developing relationships with schools and community organizations to facilitate 
the smooth implementation of the program. Overall, evaluation evidence suggests 
that OST programs hired a cadre of diverse and qualified program directors to 
launch the first year of programming under the DYCD initiative. More than half 
of Option I OST program directors reported in surveys that they had directed an 
out-of-school time program prior to the OST initiative (60 percent). In general, 
program directors were also well educated. Eighty-six percent of program 
directors had completed a four-year college degree or higher. On average, OST 
program directors reported high levels of satisfaction with their jobs and the level 
of support they received. Ninety-one percent of Option I program directors 
“agreed a lot” that they found the work at their program to be rewarding. 

Most programs hired experienced professionals to help guide the 
programs, although many programs also relied on young staff members. Eifty-six 
percent of Option I program directors reported hiring a master teacher or 
educational specialist, 62 percent of directors reported employing staff members 
with teaching certificates, and 78 percent said that at least some staff with a 
college degree. More than three-quarters (78 percent) of Option I program 
directors reported having college students on staff, and less than half (45 percent) 
employed teen staff. 

In surveys, all program directors acknowledged challenges in hiring 
program staff. Thirty-three percent of program directors reported that it was a 
major challenge to offer the competitive salaries necessary to hire qualified staff, 
and 25 percent reported that it was a major challenge to find volunteers with the 
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necessary time and expertise. The surveys permitted program directors to define 
“qualified staff’ and “necessary time and expertise” in any way they chose. 



Professional development in OST programs. OST programs provided 
professional support to their staff members through program-level supervision 
and professional development as well as through support and opportunities to 
attend professional workshops offered through the OST initiative and the provider 
organization. (DYCD provided $500,000 in technical assistance and professional 
development to programs through a contract with Partnership for After School 
Education.) Almost all Option I program directors reported holding meetings 
with their OST program staff at least once a month (98 percent). Thirty-nine 
percent of program directors reported holding meetings with their staff at least 
once a week. A majority (68 percent) of program directors also required at least 
some staff to submit lesson plans on a regular basis. More than half of program 
directors (57 percent) used a published or externally developed curriculum. 
Program directors received technical assistance most frequently in the use of OST 
Online (83 percent), and about half reported receiving technical assistance in 
program design and implementation and program management and administration 
(54 percent and 5 1 percent, respectively). 

Program policies. In the first year of the OST initiative, program 
directors worked to create program policies and procedures that ensured 
compliance with city and state regulations and that provided a foundation for a 
positive program environment. More than three-quarters of Option I program 
directors reported “to a great extent” that their program had policies in place to 
report suspicions of child abuse or neglect, ensure enough time for program 
activities, and deal with participant behavior. Among elementary-grades Option I 
program directors, 99 percent reported that policies were in place to report 
suspicions of child abuse or neglect. 



Implementation of Process and Content Features 

The DYCD OST initiative encouraged programs to offer a variety of rich 
content-based activities to support multiple domains of youth development. 
Among Option I programs serving elementary-grades programs, almost all 
offered homework help to their elementary-grades participants (98 percent), 
according to surveys of program directors. In addition, 84 percent reported 
offering many visual arts and crafts activities. Many regularly offered free time 
for physical play and unstructured time for socializing (59 percent and 57 percent, 
respectively). Middle-grades programs offered a similar set of activities, although 
organized team sports and dance/movement activities were more common in the 
middle-grades programs (reported by 73 percent and 53 percent of program 
directors). Findings suggest that programs targeting high school youth were more 
specialized and tended to be more civic-oriented, reflecting the interests of high 
school youth. 
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Participant Engagement in OST Services 



The DYCD initiative established elear standards for program partieipation 
that varied based on program option and grade level served. Option I elementary- 
grades programs were expected to serve youth for a minimum of three hours a 
day, five days a week, for 36 weeks, with a participation target of 80 percent. 
Middle-grades programs were expected to offer programming for at least eight 
hours per week for 36 weeks, with a 75 percent participation target. Programs 
serving high school youth were expected to provide a minimum of three hours of 
programming per week for 36 weeks, with a 70 percent attendance target. Option 
I elementary- and middle-grades programs were also expected to provide OST 
service for 10 hours a day on 20 days per year when schools were closed. 

Based on data from OST Online on programs that provided services for 
the full school-year and that also entered reliable participation data into the 
system, participants in Option I elementary-grades programs attended their OST 
program for an average of 3 1 1 hours during the year, compared to the expected 
average of 432 hours. Participants in middle-grades programs attended for an 
average of 154 hours, compared to the target of 216 hours. Option I high school 
participants surpassed their targeted attendance of 76 hours, attending for an 
average of 97 hours. Across all participants, a quarter of elementary- and middle- 
grades participants met the targeted number of hours (25 percent and 26 percent, 
respectively), while 39 percent of high school participants met attendance 
expectations. These calculations of attendance levels may, however, have been 
suppressed by program-level difficulties in entering activity and attendance data 
into OST Online. 



Social Development Outcomes of Youth 

Out-of-school time programs can play an important role in helping youth 
to develop the social skills with both adults and peers that they need to mature 
into successful adults themselves. In addition, by providing engaging 
programming that exposes youth to opportunities that they would not otherwise 
experience, OST programs can increase their draw for participants and their 
capacity to contribute to youths’ academic and social development. 

Overall, about half of OST participants “strongly agreed” that their 
program had exposed them to new and interesting activities in Year 1, according 
to a large sample of OST participants. Across all grade levels, 53 percent of 
youth “agreed a lot” that the program gave them a chance to do “a lot of new 
things.” In addition, more than half “agreed a lot” that the program activities 
“really got them interested” (56 percent). 
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Evaluators examined soeial interaetions in OST programs. The survey 
asked partieipants in elementary-grades programs a series of questions about their 
interaetions with their peers, and youth reported positive peer interaetions. Youth 
most frequently reported that they “had a good time playing with other kids in the 
program,” with 70 percent agreeing “a lot.” Participants also tended to “agree a 
lof ’ that they had “a lot of friends in the program” (69 percent) and that they “got 
to know other kids really well in the program” (64 percent). In general, 
participants also reported positive interactions with OST program staff members. 
Across all grade levels, 68 percent of participants “agreed a lot” that staff treated 
them with respect, and 67 percent reported that staff thought that they could learn 
new things, although only 44 percent of youth “agreed a lot” that staff always 
kept their promises. 

Overall, Option I participants across all grade levels reported relatively 
high levels of attachment to their OST program. Across all Option I participants, 
74 percent of youth “agreed a lot” that they felt safe in the OST program, and 60 
percent “agreed a lot” that they felt like they “belonged” in the program and that 
the program was “a good place to hang out.” 



Youth Content/Academic Skill Outcomes 

OST programs promote many types of skills and knowledge, depending 
on the organizational focus of their sponsoring organization and the skills of their 
staff. The evaluation asked youth to report on a series of measures of the 
academic benefits of participating in an OST project. In general, participants in 
elementary-grades programs were significantly more likely to report high levels 
of academic benefits compared to either middle-grades or high school 
participants. For example, the majority of elementary-grades participants (67 
percent) “agreed a lot” that the program helped them finish their homework more 
often, significantly more than the 58 percent of middle-grades participants and 33 
percent of high school participants. Elementary-grades youth were also 
significantly more likely to report that the program helped them feel better about 
their schoolwork, compared to middle-grades youth and high school youth (53 
percent, compared to 35 percent and 31 percent respectively). A similar pattern 
emerged in participant reports of academic self-esteem, which decreased 
significantly between each grade cohort from elementary grades to middle grades 
to high school. Students reported significantly higher levels of academic self- 
esteem on the following measures compared to both middle-grades and high 
school participants; middle-grades participants had significantly more positive 
responses on these same items compared to the high school participants. For 
example, 81 percent of youth in elementary-grades programs agreed a lot that 
they tried hard in school, compared with 68 percent of youth in middle-grades 
programs and 5 1 percent of youth in high school. 
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Associations Between Program Features and Participant 
Experiences 



An objective of the evaluation is to determine whether specific program 
characteristics are statistically associated with participants’ engagement in the 
OST program and their social development and educational outcomes. Evidence 
from Year 1 indicates that participants who attended programs with a strong 
academic focus reported more academic benefits from OST participation and 
higher academic self-esteem. (For this analysis, the evaluators created and 
employed a variable measuring program-level academic focus, based on program 
directors’ survey responses regarding activities supporting literacy-skill 
development, cognitive development, and academic achievement.) In addition, 
higher levels of arts activities were also positively associated with self-reported 
academic benefits and academic self-esteem. Positive correlations were found for 
academic self-esteem and frequency of OST attendance, especially in school- 
based programs. In contrast, analyses found a negative association between 
participant reports of the quality of their interactions with OST program staff and 
the level of academic activities offered in the program. High levels of physical 
fitness activities were negatively associated with participant reports of (1) new 
and engaging experiences in the OST program and (2) positive interactions with 
program staff. 



Systems Outcomes 

The evaluation collected data to assess the extent to which the OST 
initiative is increasing the capacity of provider organizations to deliver high- 
quality OST services, increasing the capacity of private nonprofit providers and 
public agencies to function as a coherent system, and meeting the needs of 
working parents. 

Effect on provider organizations. For the most part, the OST provider 
organizations funded in Year 1 operated with sizable budgets: 52 percent of 
executive directors reported that their annual organizational budget was more than 
$3 million, and an additional 10 percent had an annual budget of more than $2 
million. 

On average, organizations drew 59 percent of their OST budgets from 
DYCD funding. Twenty percent of organizations relied exclusively on DYCD 
funds for their out-of-school-time programming budget. 

When asked how the OST contract had affected the organizations’ 
operations, executive directors reported that participation in the DYCD OST 
initiative had had the most effect on staff opportunities for training and technical 
assistance: 74 percent responded that DYCD participation increased their 
opportunities to participate in training either “to a great extent” or “somewhat.” 
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For the majority of providers, DYCD partieipation also inereased their 
opportunities to partner with city agencies (62 percent), a public school (58 
percent), and cultural organizations (55 percent). 

Increase in system capacity. The majority of provider organizations had 
extensive experience operating out-of-school time programs prior to the DYCD 
OST initiative, according to survey responses. Eighty-seven percent of executive 
directors surveyed said that their organization had previously operated an after- 
school or out-of-school time program. Seventy percent of provider organizations 
had operated programs since at least the 1997-98 school year, although almost 
half (44 percent) of program directors reported that the initiative brought out-of- 
school time programming to their location for the first time. 

Meeting the needs of working parents. A key goal of the OST initiative 
is to support the needs of working families. Indeed, the majority of parents who 
responded to the parent survey (conducted in 15 OST programs) indicated that 
they were working parents. More than three-quarters of responding parents (81 
percent) reported that they worked at least 20 hours per week. About a third of 
parents (34 percent) responded that they were enrolled in school. 

Overall, parents were satisfied with the quality of the OST program their 
child attended. Sixty-two percent of parents rated the overall quality of their 
programs as excellent. Parents were particularly satisfied with their OST 
program’s ability to provide a safe space for students to participate in activities 
and interact with other youth. Sixty-seven percent of parents strongly agreed that 
their child was able to join activities that they would not otherwise experience, 67 
percent felt their child was safer in the out-of-school hours as a result of the 
program, and 62 percent reported that their child made new friends. 

In addition, about half of responding parents “strongly agreed” that their 
child benefited academically from participation in the OST program. Fifty-five 
percent “strongly agreed” that their child was getting the academic help he/she 
needed, 54 percent felt that their child talked to them more about what was going 
on in school, and 54 percent “strongly agreed” that their child was doing better in 
school overall as a result of participating in the after-school program. 

Parents also expressed satisfaction in terms of how well the OST program 
met their own needs. Seventy-one percent of parents “strongly agreed” that the 
program hours fit their needs, 63 percent reported that they missed less work than 
before the program, and 61 percent reported that the program made it easier for 
them to keep their job. 




Conclusions 



The evaluation’s major findings for aetion in Year 2 foeused on four areas. 
First, although OST programs suceessfully enrolled students in the initiative’s 
first year, programs seem to have struggled to maintain high participation rates. 
Some of these apparent attendance issues may actually reflect program-level 
deficiencies in keeping activity-participation data current in OST Online. Second, 
all data sources available to the evaluation confirmed that programs consistently 
provided safe and structured environments for participants to spend out-of-school 
time, a noteworthy achievement in the initiative’s start-up year. Third, OST 
programs experienced challenges in hiring program staff members who were well 
qualified to provide effective OST programming. Limited resources for staff 
compensation contributed to these challenges. Finally, evidence of differing 
participant reactions to school-based and center-based programs suggests that 
each program type has strengths relevant to the operations of programs that 
provide high-quality experiences for youth. 




1. Overview of Program Development and Goals 



This section of the report briefly describes the expectations and goals for 
the Out-of-School Time (OST) initiative, which is the subject of this evaluation. 



Development of the OST Initiative 

The New York City Department of Youth and Community Development 
(DYCD) launched services under its OST Programs for Youth in September 2005 
with the award of funds to nearly 200 community-based and other nonprofit 
organizations for the support of more than 525 programs across New York City. 
Together, DYCD and the city’s nonprofit community, working closely with the 
New York City Department of Education (DOE), extended services to 
approximately 51,000 youth across the city in the initiative’s first year, making 
OST almost certainly the nation’s largest after-school initiative. 

Although DYCD has supported programs for youth over many years, it 
began planning for the OST initiative in 2003, when it consulted with city 
agencies, youth-serving nonprofit organizations, community leaders, and private 
funders to develop an OST Program Vision and Goals statement. After reviewing 
external reactions to an early concept paper on approaches to operationalizing its 
OST vision and goals, DYCD issued its request for proposals (REP) for OST 
services in December 2004. The REP solicited offers to address five service 
options, described below, hollowing competitive review of proposals using 
selection criteria published in the REP, DYCD negotiated and awarded contracts 
under all five options by the end of summer 2005. 

Option I was designed to fund OST programs for youth in elementary, 
middle, and high schools in each of the DOE’s 10 geographically defined 
administrative subdivisions, known as regions. Option I also included 15 
“priority middle schools” in which OST programs would operate in collaboration 
with state-approved Supplemental Educational Services providers. As illustrated 
in Exhibit 1 , the program expectations for Option 1 varied by grade level served, 
with programs for younger students expected to provide more programming hours 
(and hence more comprehensive services) than programs serving older students. 

Option 11 was designed to support OST programs that would use private 
match funds to subsidize at least 30 percent of their OST budgets. These 
programs were intended to serve students of any grade level for a minimum of 
four weeks and 160 hours per year, with a maximum DYCD award of $600 per 
participant. 
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Exhibit 1 

Option I Program Parameters 





Minimum Time of Operation 






School Year 


Summer 


School Closing Days 




Maximum 






Hours/ 




Hours/ 




Hours/ 


Total 


Award per 


Program Level 


Weeks 


week 


Weeks 


week 


Days 


day 


Hours 


Participant 


Elementary grades 


















Year-round programs 


36 


15 


8 


50 


20 


10 


1,140 


$2,800 


School year programs 


36 


15 


N/A 


N/A 


20 


10 


740 


$2,000 


Middle grades 


















Year-round programs 


36 


8 


8 


50 


20 


10 


888 


$2,100 


School year programs 


36 


8 


N/A 


N/A 


20 


10 


488 


$1,300 


High school 


















School year programs 


36 


3 


N/A 


N/A 


N/A 


N/A 


108 


$540 



Option III programs were to be operated in collaboration with the 
Department of Parks and Recreation and would be offered at Parks sites. Each 
Option III program was expected to address one of three purposes: (1) Reach 
programs were to offer therapeutic recreation and educational programming for 
youth with disabilities; (2) Academic Support programs were to offer activities to 
assist participants to meet or exceed academic standards; and (3) Life Skills 
programs were to emphasis instruction to youth in how to manage their finances. 
All of these programs were expected to be in operation for 36 weeks during the 
school year. The Reach programs were required to offer a minimum of 10 hours 
of programming per week. The Academic Support and Life Skills programs were 
required to offer a minimum of six hours of programming per week for 
participants ages 6-13 and a minimum of four hours per week for participants 
ages 14-21 . DYCD’s maximum award was $2,500 per participant for the Reach 
programs and $540 per participant for the Academic Support and Life Skills 
programs. 

Option IV solicited proposals to provide technical assistance to OST 
programs. Option V solicited proposals to evaluate the overall initiative. 

The RLP stated that DYCD would give greater consideration to proposals 
for Option I and II programs that planned to serve youth in zip codes with a high 
need for OST services. DYCD identified these priority zip codes based on the 
following five criteria: the population of youth ages 6-15 years residing in the zip 
code, the youth poverty rate in the zip code, the percent of youth ages 16-19 in the 
zip code who are not in school (and not high school graduates or in the labor 
force), the number of ELL students in DOE schools in the zip code, and the 
number of single-parent families with children under 1 8 years of age in the zip 
code. Using these criteria, 25 high-need zip codes were identified for Option II 
programs. Lor purposes of Option I programs, DYCD extended the list to make 
sure that each of the 10 DOE regions had at least five targeted zip codes. 
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DYCD’s Vision and Goais for the OST Initiative 



DYCD described its OST vision as follows in the RFP (page 9): “A 
quality OST system offers safe and developmentally appropriate environments for 
children and youth when they are not in school. OST programs support the 
academic, civic, creative, social, physical, and emotional development of young 
people and serve the needs of the city’s families and their communities. 
Government, service providers, and funders are partners in supporting an 
accountable and sustainable OST system.” 

Accordingly, DYCD’s nine program goals reflect this vision (pages 9-10): 

1 . Provide a healthy, safe environment 

2. Foster high expectations for participants 

3. Foster consistent and positive relationships with adults and peers 
and a sense of community 

4. Support the needs of working families 

5 . Support healthy behavior and physical well-being 

6. Strengthen young people’s academic skills 

7. Support the exploration of interests and the development of skills 
and creativity 

8. Support youth leadership development 

9. Promote community engagement and respect for diversity 

Seen in the context of the national OST movement that has emerged and 
grown over the last ten years, these program goals cover an especially broad range 
of developmental objectives for youth participants. In particular, they are less 
narrowly academic than the goals articulated in many other OST and after-school 
program authorizations, such as the national 2D* Century Community Learning 
Centers Program. DYCD’s goals emphasize positive youth development in 
multiple domains within a context of safety and also support for the needs of 
working families. 



2. Evaluation Design and Operations 
in the First Year 

This section describes the design of the evaluation, including the theory of 
change on which it is based, and the implementation of the evaluation in its first 
year. 
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Overview of Evaluation Design 

A theory of change was developed to guide the evaluation, based on the 
OST initiative’s goals and on discussions with DYCD staff. The change theory 
provided the basis for the evaluation questions and for the evaluation design. This 
change theory, illustrated in Exhibit 2, is based on the following propositions: 

■ To achieve positive life outcomes, children and youth require 
opportunities and supports in multiple developmental domains, 
including academic, social, psychological, and physical areas. 

■ High-quality after-school programs can stimulate positive 
experiences and outcomes for children and youth in these core 
developmental domains when children and youth participate 
regularly and for more than a year. 

■ After-school programs can promote positive development through 
many types of program content (e.g., arts, academics, culture, 
sports), so long as program staff promote high levels of youth 
engagement and positive relationships among youth and adults. 

■ Community-based organizations with a commitment to youth 
development can adopt policies and practices that equip them to 
deliver OST services capable of promoting positive development 
and exposing youth to the diverse resources of their own 
immediate communities and of the city itself. 

■ Certain structural and institutional features of programs, such as 
qualified staff and appropriate group sizes, also facilitate the 
implementation of high-quality programs. 

■ To achieve the goal of sustainable, universal OST program 
coverage for all youth who desire program access, the city and the 
provider community need to develop close partnerships that are 
characterized by a commitment to both OST quality and access. 

The core task in designing the OST evaluation was to devise a framework 
that would capture information at baseline and over time that responded to the 
OST goals while also accounting for the broad diversity of OST programs and 
participants. Indeed, the sheer scale of the program posed the largest single 
challenge in crafting an appropriate evaluation design. The design that resulted 
from these deliberations employs systematic nesting of samples to permit 
generalization from the few to the many. Key features of the design are sketched 
below. 
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Exhibit 2 

Theory of Change for the Evaiuation of OST Programs 



/ \ 

Structural and 
Institutional Features 

V J 



r 






Process and Content 
Features 






J 



r 






Short-Term Outcomes 






J 



r 






Long-Term Outcomes 






J 
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From all programs in Options I through III, the evaluation is 
eolleeting the following types of data annually for three years: 

Data from OST Online, DYCD’s program information system, 
whieh was designed speeifieally for use by DYCD’s OST program 
and whieh OST programs use to reeord and maintain information 
deseribing the eharaeteristies and OST involvement of all 
partieipants, plus other program information. OST Online was 
launched in Year 1 of the initiative; additional fields will be 
required, and modifications to the system will be made to better 
meet the needs of DYCD programs, and the evaluation, in future 
years. 

Survey of program directors, which collects data on program 
goals and activities, program schedules, staff recruitment and 
qualifications, participant outreach and recruitment, participant 
needs and preferences, and linkages with participants’ schools, 
communities, and families 

Survey of executive directors of provider organizations, which 
collects data on how the OST program influences provider 
organizations in fulfilling their core missions, how OST programs 
link to other services delivered by provider organizations (if at all), 
and the cost and funding of specified elements of OST programs 

In addition to the data elements listed above, the evaluation is 
collecting data through annual participant surveys in a stratified 
random sample of 133 Option I programs. In these sites, 
evaluators are administering surveys to all participants in grades 3- 
12. This sample is structured to permit findings to be generalized 
to Option I programs as a whole and also to Option I programs at 
elementary, middle, and high school levels and to Option I 
programs that are either school-based or center-based. 

Separate from the stratified random sample of Option I programs, 
the evaluation is also collecting participant survey data annually in 
the 15 Priority Middle Schools sites (as described on page 1 of this 
report). 

In a random sample of 15 sites selected from the sites in the 
participant-survey sample, known as the in-depth sample, the 
evaluation is conducting annual site visits. During the site visits, 
evaluators conduct individual and small-group interviews, 
structured observations of program activities, and surveys of staff 
and parents. In the second and third years of the evaluation, 
evaluators will obtain educational and demographic data on 




students whose names and DOE identifieation numbers are 
available in OST Online. 

With these data sourees, the evaluation is addressing four primary researeh 
questions: 

1 . What are the eharaeteristies of the programs supported by the OST 
initiative? 

2. Who partieipates in these programs, and what are their patterns of 
attendanee? 

3. What are partieipants’ patterns of soeial and emotional growth? 

Do programs affeet partieipants’ edueational performanee and, if 
so, how? 

4. Do programs meet the eity’s needs for assistanee to working 
parents and for improvement in eommunity-level eapaeities to 
serve youth during the out-of-sehool hours and, if so, how? 

The OST evaluation is eurrently designed to eolleet and report on data 
spanning the initiative’s first three years of operations. The evaluation’s design 
reeognizes that the initiative’s first year was foeused on launehing a very large 
number of programs throughout the eity. Evaluators looked for program 
providers in Year 1 to hire needed staff, reeruit and retain an initial eohort of 
youth partieipants, provide safe environments, and deliver aetivities and serviees 
that generally responded to student and parent interests. As reported here, OST 
programs aehieved these objeetives. 

In the seeond and third years, evaluators’ expeetations will be somewhat 
higher. In partieular. Year 2 data on OST programs should provide evidenee that 
providers are aehieving at least some of the quality objeetives artieulated in 
DYCD’s 2004 Request for Proposals (REP). Year 3 should generate preliminary 
evidenee of positive early outeomes for partieipating youth. 



Evaluation Operations in Program Year 2005-06 

Evaluation start-up oeeurred at the same time as OST programs began 
operations in late summer 2005. The evaluators developed and finalized the 
evaluation design, samples, and data-eolleetion instruments in eonsultation with 
DYCD on a rolling basis starting at the beginning of the program year, with 
internal development and review timetables set to eorrespond to sehedules for 
notifying sites about the evaluation and eolleeting data. In aeeordanee with DOE 
rules, the evaluator sought and obtained DOE approval of the surveys and of the 
eonsent forms and praetiees to be used to proteet the privaey of student data. 
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Working with the evaluator, DYCD staff eontaeted OST programs to 
obtain two types of necessary research consents in Year 1: 

■ Parental research consents, using a form developed by DYCD 
and approved by DOE 

DYCD requires programs to enter each participant’s 
parent/guardian research-consent status into the appropriate field in 
OST Online. Of 32,1 17 Option I participants in grades 3-12 with 
data in OST Online, as of May 8, 2006, the parents or guardians of 
18,514 participants provided research consent, while parents or 
guardians of 904 participants denied consent. No consent data 
were recorded for 12,697 Option I participants in grades 3-12 (40 
percent of total participants in these grades). 

■ Principal research approvals, using the form provided by DOE 

DOE rules require that the principal of any host school approve in 
advance of any research activities to be conducted in that school 
and involving students or DOE employees. Accordingly, the 
evaluators, in collaboration with DYCD, sought research approval 
from principals of schools with OST programs that were either 
included in the Option I participant survey sample or were part of 
the Priority Middle Schools Program. Among the 85 school-based 
programs in the Option I participant survey sample, 78 principals 
(92 percent) granted research approval. Evaluators and DYCD 
were unable to secure consent from seven principals. Among the 
15 Priority Middle Schools, 12 principals (80 percent) granted 
research approval, and three did not. 



Year 1 Data Collection 

The findings in this report are based on data collected from the following 
sources during the first year of the OST initiative: 

■ OST Online. The evaluation analyzed patterns of enrollment and 
participation for programs and participants that had entered data 
into OST Online. This included 40,818 participants in 417 Option 
I programs, 9,139 participants in 100 Option II programs, and 
1,021 participants who attended 1 1 Option III programs during the 
2005-06 school year. Overall, 50,978 participants were enrolled in 
programs, according to OST Online. 




■ Survey of executive directors of provider organizations. The 

evaluation administered an online survey to all executive directors 
of organizations that received an OST contract, in Spring 2006. 
Findings in this report present data from 161 of 190 executive 
directors (approximately 85 percent response rate). 

■ Survey of program directors. In Spring 2006, the evaluation 
administered an online survey to directors of all OST programs, in 
Options I-III. Data represent the responses of 483 program 
directors from 543 programs,' for a response rate of approximately 
89 percent. Surveys were completed by 393 Option I program 
directors, 80 Option II directors, and 10 Option III directors. 

■ Survey of participants. Paper surveys were administered to OST 
participants in grades 3-12 who attended the 133 randomly 
selected programs in the evaluation sample and who had parental 
consent to participate in the evaluation. Survey data presented in 
this report are based on data from a total of 3,614 participant 
surveys from 95 programs (71 percent of the sampled programs), 
including 1,81 1 surveys from 48 elementary-grades programs, 
1,047 surveys from 23 middle-grades programs, and 756 surveys 
from 24 high school programs. 

In addition, surveys were administered to consented participants in 
the Priority Middle School programs. Data in Appendix A of this 
report represent the responses of 562 students in 1 1 of 15 Priority 
Middle School programs. 

■ Survey of program staff. Paper surveys were administered to 
staff members in the 15 randomly selected in-depth study 
programs in Spring 2006. Staff-survey data in this report represent 
findings from 114 staff members in 12 programs, including 71 
elementary-grades programs, 28 middle-grades programs, and 15 
high school programs. The number of surveys received per 
program ranged from 2 to 16. 

■ Survey of parents. Paper surveys were administered to parents of 
OST participants in the 15 Option I in-depth study programs in 
Spring 2006. Parent-survey data in this report represent the 
responses of a total of 283 parents in 12 programs. The number of 
parent surveys received from each program ranged from 3 to 78. 



* Readers may note that the 543 programs is more than the 528 programs with participation data 
in OST Online, as reported above. The 543 programs were identified from DYCD’s master list of 
programs, a few of which had not entered data into OST Online by the end of the program year. 
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■ Site visit observation data. Evaluators conducted two-day site 

visits to eaeh of the 15 Option I in-depth study programs in Spring 
2006. These visits ineluded interviews with the program direetor, 
program staff, partieipants, and, in many oases, parents. Data in 
this report also include analyses of structured activity observations 
oonduoted during these visits. 

In general, analyses in this report are presented separately for programs in 
OST Options I, II, and III. Program director and participant survey responses for 
Priority Middle Sohool programs are analyzed separately in Appendix A. Within 
Option I, many analyses oompare oenter-based and sohool-based programs as well 
as elementary-, middle-, and high sohool programs. Appropriate tests were 
conducted to determine whether the differences between program types were 
statistioally signifioant, inoluding Chi-Square and ANOVA tests with Bonferroni 
post-hoe tests to distinguish whether, for example, elementary-grades programs 
varied significantly from middle-grades programs and/or high sohool programs. 
The findings presented in this report focus on areas in whioh statistioally 
signifioant differenoes were found between groups. 



3. Scope and Extent of OST First-Year 
Programming 

In Fall 2005, DYCD issued oontraots for 557 OST programs to oonduct 
OST aotivities in the 2005-06 program year, inoluding 430 Option I programs, 

114 Option II programs, and 13 Option III programs. Evaluators used available 
data entered in OST Online to determine program enrollment in Year 1 . In the 
first year of implementation, some programs struggled to aooess and enter data in 
OST Online. As a result, it is possible that some programs that were in operation 
are not ineluded in the enrollment analyses beeause they did not enter any 
partieipation data in OST Online. 

By the end of the school year, aeeording to OST Online data, 528 
programs were launehed and had partieipation data available, with 417 sites 
operating Option I programs, 100 operating Option II programs, and 1 1 operating 
Option III programs. Among all sites, 314 were based in New York City sehools, 
and 214 were based in other loeations (and are known as center-based programs). 

Aeross these 528 programs, as shown in Exhibit 3, DYCD awarded 
eontracts that were intended to serve 15,462 youth in elementary-grades 
programs, 9,659 youth in middle-grades programs, 1 1,512 youth in high sohool 
programs, and 7,183 youth in programs serving multiple grade levels, for a total 
of 43,816 youth, aeeording to DYCD’s master list of programs. As desoribed in 
programs’ entries in OST Online, these programs aotually served a total of 50,978 
students from September 2005 through June 2006. Option I programs aeross all 
grade levels served more students than were speeified in their eontraets, with 
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programs funded to serve 31,668 students and aetually serving 40,818 students. 
Programs supported under Options 11 and 111 were awarded funds to support 
enrollments of 10,673 and 1,475 students, respeetively, and served fewer students 
overall than their award enrollments. 



Exhibit 3 

Targeted Enroiiment and Actuai Number of Students Served, 
by Option and Grade Levei 



Grade Level 


Option 1 


Option II 


Option III 


All Programs 


Targeted 

Enrollment 


Students 

Served 


Targeted 

Enrollment 


Students 

Served 


Targeted 

Enrollment 


Students 

Served 


Targeted 

Enrollment 


Students 

Served 


Elementary 


13,707 


18,225 


1,755 


1,632 




N/A 


15,462 


19,857 


Middle 


9,340 


14,261 


319 


235 


N/A 


N/A 


9,659 


14,496 


High 


8,621 


8,332 


2,891 


2,540 




N/A 


11,512 


10,872 


Multiple 


N/A 


N/A 


5,708 


4,732 


1,475 


1,021 


7,183 


5,753 


Total 


31,668 


40,818 


10,673 


9,139 


1,475 


1,021 


43,816 


50,978 



Many OST programs may have enrolled partieipants who were not 
direetly funded by OST monies but reeeived the same serviees and aetivities as 
partieipants who were OST-funded, beeause the programs were able to use their 
OST eontraets to leverage additional funds. Aeross all options, 84 pereent of 
students entered in OST Online were designated as OST-funded partieipants; this 
was true for 81 pereent of Option 1 partieipants, 97 pereent of Option 11 
partieipants, and 93 pereent of Option 111 partieipants. 

Ineluding all partieipants, 74 pereent of Option 1 programs met or 
exeeeded their enrollment targets. Forty-eight pereent of Option 11 programs and 
9 pereent of Option 111 programs met or exeeeded their enrollment targets. Aeross 
all options, 87 pereent of elementary-grades programs met or exeeeded their 
enrollment target, eompared with 73 pereent of middle-grades programs, 53 
pereent of high sehool programs, and 36 pereent of programs serving multiple 
grade levels. 

Although OST Online does not distinguish between enrollees who replaee 
departed students and other new enrollees, evaluators were able to determine the 
pereent of students who were enrolled for either only the fall months or only the 
spring months. In Option 1 programs, 10 pereent of students attended in the fall 
only, and 22 pereent attended in the spring only. Comparable figures were 9 
pereent and 21 pereent for Option 11 programs and 10 pereent and 49 pereent for 
Option III programs. These numbers suggest that, while some mid-year enrollees 
were replaeements for others, it is also likely that programs eontinued to maintain 
open enrollment to meet their targeted enrollment throughout the year. 
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DYCD awarded over $44 million to the 528 programs, with an average 
award of $84,000 per program and a median award of $73,000. Award amounts 
ranged from $3,000 to $340,000. DYCD awarded these contraets to 187 provider 
organizations.^ Many provider organizations reeeived more than one award, as 
shown in Exhibit 4. Nine provider organizations reeeived 10 or more contracts, 
with total OST funding per organization ranging from $210,000 to $2.3 million. 



Exhibit 4 

Provider Organizations with 10 or More OST Programs 



Provider 


Number of 
Programs 


Sports and Arts in Schools Foundation 


28 


The Children’s Aid Society 


25 


Police Athletic League, Inc. 


20 


The After-School Corporation 


14 


Church Avenue Merchants Block Association, Inc. 


12 


New York Junior Tennis League 


12 


Madison Square Boys & Girls Club 


10 


Global Kids, Inc. 


10 


Big Brothers/Big Sisters of NYC 


10 



The REP specified different numbers of hours that Option I elementary, 
middle, and high school programs would operate per week, as shown in Exhibit 1 . 
The stated expectations in the REP for school-year programs were 15 hours, 8 
hours, and 3 hours, respectively, with a further expectation that all elementary and 
middle-grades programs would be open for 10 hours a day on 20 days during the 
school year when the schools were closed due to holidays and to winter, mid- 
winter, and spring recess periods. Prom September 2005 through June 2006, 
according to OST Online, the 528 OST programs were open for periods that 
ranged from 12 to 289 days. Delays in program start-up likely account for the 
reports of low numbers of days of operation, with some programs not beginning 
operation until April. The average program service period was 162 days, and the 
median was 188 days. As expected, programs serving younger students were 
open for more days on average than were programs serving older students. 
Elementary-grades programs were open for periods that ranged from 30 to 251 
days, with an average of 193 days (median of 199 days), middle-grades programs 
were open between 19 and 251 days, with an average of 172 days (median of 186 
days), and high school programs were open for 12 to 260 days, with an average of 
131 days (median of 136 days). Programs serving multiple grade levels were 



^ The earlier discussion of Year 1 data collection reports that executive director surveys were 
administered to directors of 190 provider organizations. These organizations were identified from 
DYCD’s master list. Only 187 organizations had programs with program data in OST Online. 
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open between 13 and 289 days, with an average of 123 days (median of 129 
days). OST Online data indieate that: 74 pereent of programs aeross elementary- 
and middle-grade levels were open every month from September through June; 23 
pereent were open six to nine months; and 2 pereent were open for five months or 
less during this time period. 

OST programs were distributed across each of the five New York City 
boroughs, as shown in Exhibit 5. Brooklyn hosted the most Option 1 OST 
programs (153), while Manhattan had the most Option II programs (39). Each of 
the five boroughs hosted at least one of the 1 1 Option III programs. The majority 
(275) of Option I programs were school-based, whereas the majority of Option II 
programs were center-based (61). All Option III programs were center-based. 
Option I programs were fairly evenly distributed across the DOE regions, ranging 
from 37 programs in Regions 1, 4, and 10 to 48 programs in Region 6. By far, the 
largest number of Option II programs was in Region 9 with 33 programs. No 
Option III programs operated in Regions 5, 6, or 9, while there were three Option 
III programs each in Regions 8 and 10. 

As described earlier, DYCD identified a series of target zip codes for 
Option I and II programs, in order to increase the availability of OST services in 
high-need areas. Overall, out of the 528 OST programs with data as of June 2006, 
297 programs (56 percent) operated in OST target zip codes and served 28,1 16 
students; 267 of these programs were funded under Option I, and 30 were funded 
under Option II. 

Option I programs most frequently served elementary-grades students, 
with 170 programs serving 18,225 students. Most Option II programs served high 
school students or students across multiple grade levels, with 37 programs serving 
2,540 high school students and 42 programs serving 4,732 students across 
multiple grade levels. All Option III programs served students across multiple 
grade levels. The 15 Priority Middle School Programs for which data were 
available provided OST services to 2,394 students. 
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Exhibit 5 

Number of OST Programs and Participants, by Option 





Option 1 


Option II 


Option III 


All Programs 




Programs 


Participants 


Programs 


Participants 


Programs 


Participants 


Programs 


Participants 




{n=417) 


(n=40,818) 


(n=100) 


{n=9, 139) 


{n=11) 


(n=1,021) 


(n=528) 


(n=50,978) 


Borough 

Brooklyn 


153 


13,307 


21 


2,521 


3 


243 


177 


16,071 


Bronx 


92 


10,850 


22 


1,634 


2 


147 


116 


12,631 


Manhattan 


68 


5,799 


39 


3,020 


3 


206 


110 


9,025 


Queens 


85 


8,709 


15 


1,884 


2 


111 


102 


10,770 


Staten Island 


19 


2,153 


3 


80 


1 


248 


23 


2,481 


Program Location 

School 


275 


31,797 


39 


3,250 


0 


0 


314 


35,047 


Center 


142 


9,021 


61 


5,889 


11 


1,021 


214 


15,931 


DOE Region 

Region 1 


37 


4,420 


7 


640 


1 


117 


45 


5,177 


Region 2 


42 


5,022 


8 


741 


1 


30 


51 


5,793 


Region 3 


43 


4,313 


7 


826 


1 


170 


51 


5,309 


Region 4 


37 


3,954 


7 


460 


1 


7 


45 


4,421 


Region 5 


42 


3,723 


8 


1,282 


0 


0 


50 


5,005 


Region 6 


48 


3,417 


4 


386 


0 


0 


52 


3,803 


Region 7 


47 


5,069 


5 


476 


1 


248 


53 


5,793 


Region 8 


40 


3,693 


7 


1,033 


3 


243 


50 


4,969 


Region 9 


44 


4,449 


33 


2,568 


0 


0 


77 


7,017 


Region 10 


37 


2,758 


14 


727 


3 


206 


54 


3,691 


Schooi Level 

Elementary 


170 


18,225 


13 


1,632 


N/A 


N/A 


183 


19,857 


Middle 


124 


14,261 


8 


235 


N/A 


N/A 


132 


14,496 


High 


123 


8,332 


37 


2,540 


N/A 


N/A 


160 


10,872 


Multiple 


N/A 


N/A 


42 


4,732 


11 


1,021 


53 


5,753 


Target Zip Codes 


267 


26,072 


30 


2,044 


N/A 


N/A 


297 


28,116 


Priority Middle 
School 


15 


2,394 


N/A 


N/A 


N/A 


N/A 


15 


2,394 



4. Demographic Characteristics of Participants 

This section describes the demographic characteristics of participants, as 
reported in OST Online by 528 programs. Evaluators examined the grade level, 
gender, race/ethnicity, and public assistance status of participants. The 
completeness of these demographic data in OST Online varied considerably. For 
example, 48,876 of 50,978 participants (96 percent) across all options had a grade 
level entered in OST Online, but only 26,674 (55 percent) had race/ethnicity data. 



Grade Level of Participants 

The majority of participants enrolled in Option I programs were in the 
elementary grades (44 percent) or middle grades (36 percent), as shown in Exhibit 
6. Twenty percent of Option I participants were in grades 9-12. In contrast. 
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Option II and III programs were more likely to enroll older participants. In 
Option II, 41 percent of participating youth were in high school, with another 44 
percent in the elementary grades and only 16 percent in the middle grades. In 
Option III, almost half of enrolled youth were in high school (41 percent), with 34 
percent of participants in the elementary grades and about a quarter in the middle 
grades. 



Exhibit 6 

Demographic Characteristics of Participants, 
by Option (in percents) 





Option 1 


Option ii 


Option iii 


Aii 

Programs 


Total Number of 
Enrolled Participants 


N=40,818 


n=9, 139 


n=1,021 


n^50,978 


Grade Span 


n=39,033 


n=8,822 


n=1,021 


n=48,876 


K-5 


44 


44 


34 


44 


6-8 


36 


16 


26 


32 


9-12 


20 


41 


41 


24 


Gender 


n=30,714 


n=5,587 


n=675 


n=36,976 


Male 


50 


49 


56 


50 


Female 


50 


51 


44 


50 


Race/ethnIcity 


n=22,417 


n=3,655 


n^602 


n=26,674 


American Indian 


1 


0 


0 


1 


Asian 


10 


11 


5 


10 


African American 


36 


33 


16 


35 


Hispanic/Latino 


39 


42 


49 


39 


Pacific Islander 


0 


0 


0 


0 


White (non-Hispanic) 


9 


9 


25 


9 


Other 


5 


6 


5 


5 



Center-based programs were more likely to enroll older participants than 
were school-based programs, as shown in Exhibit 7. In center-based programs, 
almost half of enrolled youth were in high school, with another 1 8 percent in the 
middle grades and 35 percent in the elementary grades. In contrast, school-based 
programs enrolled half of their enrolled youth in the elementary grades (48 
percent), 39 percent in the middle grades, and 14 percent in high school. 
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Exhibit 7 

Demographic Characteristics of Participants, 
by Program Location (in percents) 





Center-based 


School-based 


All Programs 


Total Number of Enrolled 


n=15,931 


n=35,047 


n=50,978 


Participants 


Grade Span 


n=1 5,159 


n=33,717 


n=48,876 


K-5 


35 


48 


44 


6-8 


18 


39 


32 


9-12 


47 


14 


24 


Gender 


n=11,108 


n=25,868 


n=36,976 


Male 


50 


50 


50 


Female 


50 


50 


50 


Race/ethnicity 


n=8,163 


n=18,511 


n=26,674 


American Indian 


0 


1 


1 


Asian 


3 


13 


10 


African American 


40 


33 


35 


Hispanic/Latino 


38 


40 


39 


Pacific Islander 


0 


0 


0 


White (non-Hispanic) 


13 


8 


9 


Other 


5 


6 


5 



Gender of Participants 

OST programs in Options I and II served approximately equal numbers of 
boys and girls, as seen in Exhibit 6. Fifty pereent of youth in Option I programs 
were male, as were 49 pereent of youth in Option II. However, Option III 
programs enrolled substantially more boys than girls (56 percent, compared with 
44 percent). 

Within Option I, programs serving elementary-grades students enrolled 
more girls than boys (52 percent compared to 48 percent), as shown in Exhibit 8. 
In contrast, programs serving middle-grades and high school participants enrolled 
slightly higher proportions of boys (53 percent and 51 percent of participants, 
respectively). 
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Exhibit 8 

Demographic Characteristics of Option I Participants, 
by Grade Levei (in percents) 





Option 1 




Elementary 


Middle 


High 


All 


Total Number of Enrolled 
Participants 


n=1 8,225 


n=14,261 


n=8,332 


n=40,818 


Gender 


n=14,451 


n= 10,586 


n=5,677 


n=30,714 


Male 


48 


53 


51 


50 


Female 


52 


47 


49 


50 


Race/ethnicity 


n=10,538 


n=7,913 


n=3,966 


n=22,417 


American Indian 


1 


1 


1 


1 


Asian 


12 


8 


10 


10 


African American 


32 


38 


43 


36 


Hispanic/Latino 


43 


39 


27 


39 


Pacific Islander 


0 


0 


1 


0 


White (non-Hispanic) 


8 


9 


13 


9 


Other 


5 


6 


7 


5 



Center- and sehool-based OST programs served roughly equal numbers of 
boys and girls, as seen in Exhibit 7, as did Priority Middle Sehool programs, as 
seen in Exhibit 9. 



Exhibit 9 

Demographic Characteristics of Priority Middie Schooi Participants 

(in percents) 





Priority Middle Schools 


Total Number of Enrolled Participants 


n=2,394 


Gender 


n=1,631 


Male 


51 


Female 


50 


Race/ethnicity 


n=1,220 


American Indian 


1 


Asian 


6 


African American 


45 


Hispanic/Latino 


38 


Pacific Islander 


1 


White (non-Hispanic) 


2 


Other 


8 
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Race/ethnicity of Participants 

OST programs in Year 1 served large numbers of partieipants from 
minority groups. Aeross all OST options, Hispanic/Latino partieipants were the 
largest group served (39 pereent of partieipants in Option I programs, 42 pereent 
in Option II, and 49 pereent in Option III), as seen in Exhibit 6. 

In Options I and II, Afriean-Ameriean youth were the seeond largest group 
served (36 percent and 33 percent, respectively). In Option III, however, only 16 
percent of enrolled participants were African-American. Instead, the second 
largest demographic group served among Option III programs was white, non- 
Hispanic youth (25 percent of participants). In contrast, white youth made up 
only 9 percent of the participant population in both Options I and II. 

Among Option I programs, programs targeting high school youth enrolled 
a higher percent of African-American participants than did elementary- and 
middle-grades programs (43 percent, compared to 32 percent and 38 percent, 
respectively), as shown in Exhibit 8. High school programs also had a lower 
proportion of Hispanic/Eatino participants than did programs serving younger 
students (27 percent, compared to 43 percent in elementary programs and 39 
percent in middle-grades programs). High school programs served a slightly 
higher percent of white students (13 percent, compared to 8 and 9 percent, 
respectively). 

Center-based programs enrolled a higher percent of African-American 
participants than did school-based programs (40 percent, compared to 33 percent), 
as shown in Exhibit 7. Center-based programs enrolled a slightly lower 
proportion of Hispanic/Eatino participants than did school-based programs (38 
percent, compared to 40 percent). Center-based programs also served a lower 
percent of Asian students (3 percent, compared to 13 percent) and a higher 
percent of white students (13 percent, compared to 8 percent). 

Among Priority Middle School programs, African-American youth were 
the largest group served, followed by Hispanic/Eatino youth (45 percent and 38 
percent, respectively), as shown in Exhibit 9. 



5. Structural and Institutional Features 
of Option I Programs 

In the first year of implementation of the OST initiative, programs strived 
to establish the program components and policies that would enable them to 
deliver high-quality services to participants. See Appendix B for technical details 
about the evaluation’s analyses of program features. 
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Program Objectives 



Reflecting the goals of the OST initiative, Option I program directors 
reported a broad spectrum of objectives. Option I program directors almost 
unanimously reported that providing a safe environment for youth was a major 
objective of their OST program (98 percent). In addition, more than three- 
quarters of Option I program directors identified the following as major objectives 
of their program: 

■ Help youth develop socially (95 percent) 

■ Provide youth with positive adult guidance and/or mentors (91 
percent) 

■ Promote respect for diversity among youth (88 percent) 

■ Help youth improve their academic performance (88 percent) 

■ Help provide health/well-being/life skills development (83 percent) 

■ Provide hands-on academic enrichment activities (8 1 percent) 

■ Provide opportunities for cultural enrichment (81 percent) 

■ Provide recreational opportunities (80 percent) 

Interviews with program directors revealed more subtly intertwined goals 
than suggested by these distinct objectives. Program directors spoke of creating 
safe environments in which youth of all ages could develop academic, social, and 
life skills that would foster a love of learning. For example, one elementary- 
grades program director said that “we encourage a love of learning by 
participating in engaging activities that are different from school but support day- 
school learning.” A high school program director noted that she developed 
activities targeted toward at-risk participants that would “inspire students’ 
attachment to school, commitment to one another, and expand their skills.” 
Another director commented, “We want the kids to have fun. We don’t want 
them to think it’s a learning experience. In the school day, it is books, books, 
books. We don’t want them to feel like they have to do [activities] the ‘right’ 
way. We want them to enjoy it and have fun so they will continue to come.” 



Strategies for Participant Recruitment 

OST Option I program directors overwhelmingly reported in surveys that 
they offered open enrollment to all youth who were interested in attending the 
program (91 percent), as shown in Exhibit 10. In addition, more than half (59 
percent) of Option I directors reported that they recruited youth who were 
recommended by school-day teachers. Programs targeting younger students 
differed significantly from programs serving older participants in the ways in 
which they targeted participants. Notably, elementary-grades programs were 
significantly more likely than high school programs to seek out youth who scored 
“below proficient” on assessments (43 percent, compared to 27 percent), 
according to program directors’ survey responses. Both elementary- and middle- 
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grades programs were significantly more likely than high school programs to 
recruit youth who were identified by their school as needing special assistance in 
reading and/or math (48 percent and 50 percent, compared to 30 percent). 
Similarly, high school programs were significantly less likely to target youth who 
received free- or reduced- price lunch than were elementary- or middle-grades 
programs (21 percent, compared to 45 percent and 40 percent, respectively), or to 
target youth with siblings attending the program (25 percent, compared to 59 
percent and 49 percent). Elementary-grades programs were also more likely to 
target youth who were English-language learners than were high school programs 
(43 percent, compared to 19 percent). 



Exhibit 10 

Types of Youth Targeted by Option I Programs for OST Services, 
According to Program Directors (n=390) 



We have open enrollment for all interested youth 

Youth who are recommended by school-day teachers 
or counselors 

Youth with siblings already attending the program 

Youth identified by their school as needing special 
assistance in reading and/or math 

Youth w ho are referred through our organization 

Youth w ho participate in other programs sponsored by 
our organization 

Youth who scored “below proficient” on city or state 
assessments 

Youth w ho are eligible to receive free- or reduced- 
priced lunch 

Youth w ho are English-language learners 

0 20 40 60 80 too 

Percent of program directors 




Program directors of school-based programs reported recruitment 
approaches that were significantly different than those used by directors of center- 
based programs. As might be expected, school-based programs were significantly 
more likely to seek to serve youth recommended by school-day teachers or 
counselors than were center-based programs (63 percent, compared to 49 
percent). School-based programs were also more likely to seek to serve youth 
identified by their school as needing special assistance in reading and/or math (46 
percent, compared to 36 percent). Reflecting their location and structures, center- 
based programs were more likely than school-based programs to serve youth who 
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participated in other programs sponsored by the organization (45 pereent, 
eompared to 34 pereent). 



Program Director Qualifications and Supports 

The role of the program direetor was key to the operations of first-year 
OST programs. OST program direetors were typieally responsible for the day-to- 
day management of the program, ineluding supervising staff, developing program 
aetivities, reeruiting partieipants, and developing relationships with sehools or 
eommunity organizations to faeilitate the smooth implementation of the program. 
Overall, evaluation evidenee suggests that OST programs hired a cadre of diverse 
and qualified program direetors to launch the first year of programming under the 
DYCD initiative. 

Program director background and qualifications. More than half of 
Option I OST program direetors reported that they had experienee as a director of 
an out-of-school time program prior to the OST initiative (60 pereent). In 
addition, more than half reported experienee as an edueational or youth-serviee 
professional sueh as a camp counselor/leader (56 percent), as a reereation, youth, 
or ehild eare worker (55 pereent), and as a staff member in an OST program (54 
pereent). Program direetor experienee did not vary by the grade level served by 
the program. However, signifieantly more program direetors in eenter-based 
programs than in sehool-based programs reported having experienee as a program 
direetor in an out-of-school time program (69 pereent, eompared to 55 pereent) 
and as a program staff member (62 pereent, eompared to 5 1 pereent). 

In general, program direetors also had high levels of edueation, as shown 
in Exhibit 1 1 . Eighty-six pereent of program direetors had eompleted a four-year 
eollege degree or higher. Eorty-four pereent had a master’s degree or higher. 

Only 19 pereent of program directors reported that they were eertified to teaeh. 

Overall, program direetors refleeted partieipants demographieally. Eorty- 
three pereent of direetors deseribed themselves as Afriean-Ameriean, 26 pereent 
as white, and 22 pereent as Hispanie/Eatino. Sixty-six pereent of program 
direetors were female. 
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Exhibit 11 

Characteristics of Option I Program Directors 





Percent of Program Directors 


Highest Education Levei Compieted (n=389) 




Less than high school 


>1 


High school or GED 


1 


Some college, other classes/training not related to a degree 


8 


Completed two-year college degree 


5 


Completed four-year college degree 


26 


Some graduate work 


16 


Master’s degree or higher 


44 


Teacher Certification (n=387) 




Certified 


19 


Race/Ethnicity (n=380) 




Black (Not Hispanic) 


43 


Hispanic/Latino 


22 


Asian or Pacific Islander 


5 


Native American or Alaskan Native 


0 


White (Not Hispanic) 


26 


Other 


5 


Gender (n=390) 




Male 


34 


Female 


66 



Supports for program directors. The salaries reported by Option I 
program direetors varied widely, ranging from below $30,000 to $55,000 or 
above per year. As illustrated in Exhibit 12, a total of 23 pereent of direetors 
reported making less than $35,000 per year, and 24 pereent made more than 
$50,000 per year. Nine pereent of program direetors reported that they earned an 
hourly wage; these wages ranged from a low of $14 per hour to a high of $39.83 
per hour, with an average hourly wage of $22.82 per hour paid to those direetors 
whose pay was eomputed on an hourly basis. 
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Exhibit 12 

Seif-Reported Saiaries of Option I Program Directors (n=370) 





Percent of Program Directors 


Salary 

Below $30,000 


17 


Between $30,000 and $34,999 


6 


Between $35,000 and $39,999 


10 


Between $40,000 and $44,999 


24 


Between $45,000 and $49,999 


10 


Between $50,000 and $54,999 


11 


$55,000 or above 


13 


Hourly wage 


9 



Executive directors of provider organizations reported offering a variety of 
benefits to their program directors, as shown in Exhibit 13, which contrasted with 
the more limited benefits available to part-time OST staff (discussed in a later 
section). Most providers offered their program directors paid time off for 
vacation and illness, as well as paid training and professional development 
opportunities (82 percent each). Program directors were also paid to attend staff 
meetings and conferences in 81 percent of programs. More than three-quarters of 
providers (77 percent) offered their program directors health insurance, while 
more than half offered a retirement savings plan (59 percent.) 



Exhibit 13 

Benefits Offered to Program Directors and Other Program Staff 
as Reported by Executive Directors (n=159) 

Paid time off for vacation and sick ieave 
Paid training or professionai development 
Paid attendance at staff meetings and conferences 
Health insurance 
Retirement savings plan 
Tuition reimbursement 
Other 

0 20 40 60 80 100 

Percent of executive directors reporting 

□ Other Program staff ■ Program director 
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Program director job satisfaction. Program directors reported high levels 
of satisfaction with their jobs and the level of support they received. As shown in 
Exhibit 14, 91 percent of Option I program directors “agreed a lot” that they 
found the work rewarding at their program, 88 percent reported that they enjoyed 
working at their program, and 77 percent of program directors reported that they 
got the support and feedback they needed from their supervisor. Job satisfaction 
among program directors did not vary by the grade level served by the program or 
by whether the program was center-based or school-based. 



Exhibit 14 

Seif-Reported Job Satisfaction of Option I Program Directors (n=384) 



I find the w ork here rew arding 



I enjoy w orking here 

I get the support and feedback I need from my 
supervisor 

i have the materiais i need to do a good job 



i have the space i need to do a good job 




Percent of program directors who "agreed a lot" 



Program Staff Quaiifications and Supports 

Program staffing patterns. The number of paid program staff employed 
by OST programs varied, with nearly half of Option I program directors reporting 
that they had 10 or fewer paid staff members (45 percent). Another 44 percent of 
program directors reported having between 1 1 and 25 paid staff members. 
Programs were fairly evenly split on whether they relied on volunteers to 
supplement paid staffing; 49 percent of program directors reported no volunteer 
staff members, and 47 percent reported having between one and ten volunteers on 
staff. 



Most programs hired experienced professionals to help guide the 
programs, although many programs also employed young staff members. Fifty- 
six percent of program directors reported hiring a master teacher or educational 
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specialist; 62 percent of Option I program directors reported employing staff 
members with teaching certificates. More than three-quarters (78 percent) of 
Option I program directors reported having college students on staff, and the same 
percent of directors reported that at least some staff had a college degree. Nearly 
half (45 percent) employed teen staff. 

Program directors acknowledged facing challenges in hiring program 
staff, as shown in Exhibit 15. All program directors reported that they were 
challenged to find qualified staff to hire, with nearly half (48 percent) reporting 
this as a major challenge. Sixty-eight percent of program directors reported at 
least some challenges in offering the competitive salaries necessary to hire 
qualified staff, and 65 percent reported challenges finding volunteers with the 
available time and necessary expertise to work in the program. In addition, 60 
percent of Option I program directors said that it was a challenge to pay potential 
staff for enough hours to attract them. All of these issues were considered to be a 
“major challenge” by at least a quarter of program directors. (The surveys 
allowed program directors to define “qualified staff’ and “necessary time and 
expertise” in any way they chose.) 

In interviews, program directors described strategies they used to recruit 
and hire the best possible staff in a challenging environment. For example, one 
program director asked teen staff candidates to submit their report cards and to 
write an essay, in order to test their ability to provide support for middle-grades 
participants. “I want to see how they can help the kids. . .They have to be good in 
literacy and math.” Another described a preference for recruiting college students 
studying education. “If I go for an education major, they are not here just for the 
job; they are here because that is want they want to do. Part of the salary is their 
own learning; it is not just to benefit us, but to benefit them because that is what 
they are learning.” In one instance, a program develops future staff members 
through the program itself, hiring former participants who have a strong 
connection to the program. “We want [our high school kids] to be our group 
leaders in the future. . .Let’s get kids from the community, train them, and this will 
be their part-time job while they’re in college. There’s method to our madness!” 
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Exhibit 15 

Staff Recruitment Chaiienges 
Reported by Option I Program Directors (n=392) 



We cannot find qualified staff to hire 

We cannot afford to offer the conpetitive 
salaries necessary to hire qualified staff 
We cannot find volunteers w ith time and 
expertise needed to help in our program 
We cannot afford to offer potential staff enough 
hours of paid employment 
Volunteers are not available on a reliable 
schedule 

There are limited professional development 
opportunities for staff 

Staff do not have the skills to w ork w ith English- 
language learners 

Staff do not come to w ork on a reliable schedule 

0 20 40 60 80 100 

Percent who reported a "major challenge" or a "minor challenge" 

□ A major challenge □ A minor challenge 
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Supports for program staff Staff salaries reflect both the prior 
experience of staff members, and the constraints directors describe in being able 
to offer a high wage. As shown in Exhibit 16, program directors most frequently 
reported an average staff wage of $1 1 to $15.99 per hour (41 percent of 
programs); 20 percent of programs reported a lower average staff wage of $6 to 
$10.99 per hour. 

Part-time OST program staff receive few fringe benefits as part of their 
employment. The main benefits as reported by executive directors of provider 
organizations were paid training or professional development and paid attendance 
at staff meetings and conferences (60 percent and 58 percent, respectively). Only 
19 percent offered paid time off, 15 percent offered health insurance, and 10 
percent offered a retirement savings plan, as illustrated above in Exhibit 13. 
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Exhibit 16 

Average Saiaries of Option [ Program Staff, According to 
Program Directors (n=393) 

Annual stipend 
$31 per hour or higher 
Between $26 and $30.00 
Betw een $21 and $25.99 per hour 
Betw een $1 6 and $20.99 per hour 
Betw een $1 1 and $1 5.99 per hour 
Between $6 and $10.99 per hour 
$0 - All staff volunteer 

0 20 40 60 80 100 

Percent who reported average staff salary 




Technicai Assistance and Supervision 

Through the OST initiative, DYCD eontracted with the Partnership for 
Afterschool Education (PASE) to provide professional development and technical 
assistance opportunities for program providers in 2005-06. According to the OST 
REP, technical assistance and professional development delivered by DY CD or its 
contractors were of two types: (1) mandatory training that all OST direct service 
providers were required to attend regarding the purposes and operations of OST 
Online, and (2) supplemental workshops and trainings that all OST direct service 
providers could attend on a voluntary basis. 

In addition, OST programs provided professional support to their staff 
members in a variety of ways, including program-level supervision and support 
and opportunities to attend technical assistance workshops offered through the 
OST initiative as well as through the provider organization. 

Almost all Option I program directors reported holding meetings with 
their OST program staff at least once a month (98 percent). Thirty-nine percent 
of program directors reported holding meetings with their staff at least once a 
week. Two-thirds of program directors (68 percent) also required at least some 
staff to submit lesson plans for review on a regular basis. More than half of 
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program directors (57 percent) used a published or externally developed 
curriculum to guide their activities. 

In interviews, program directors described ways in which their program 
and provider organization provided training and support to staff. For example, 
one program “put out a survey agency-wide to see what kind of training the after- 
school staff might want. We ask them strengths and weaknesses; we ask them 
what they need to do a better job.” Another program director described its 
strategy for training staff to advance the academic goals of the OST program. “In 
math, we try to teach [staff] how they teach math in school. . .so we retrain [our 
group leaders] to focus not always on [getting the] right answer but on the process 
of how to get there. The skill developers are trained. . .math specialists who are 
New York state certihed teachers. They take the curriculum and work with group 
leaders to implement it.” 

In addition, staff discussed the frequency and content of program staff 
meetings. For example, in one program, staff reported that they attended 
mandatory staff meetings once every two weeks after program hours. “In every 
single meeting, we would go through what is going on in class, certain kids, and 
their behavior and progress, and we share information and ideas,” said one staff 
member. 

Program directors reported that they used some of the training 
opportunities available through the OST initiative for themselves and their staff. 
The majority of Option I program directors reported receiving technical assistance 
in using OST Online (83 percent), and about half reported receiving technical 
assistance in program design and implementation and in program management 
and administration (54 percent and 5 1 percent, respectively), as shown in Exhibit 
17. In interviews, program directors typically described sending a few staff to 
training workshops offered through the DYCD OST initiative and asking those 
staff to share the information learned with their colleagues. “When someone goes 
to a good [workshop], we have them train everyone else.” 
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Exhibit 17 

Participation in Technicai Assistance and Training, 
as Reported by Option I Program Directors (n=361) 



Technical assistance in using OST Online 

Technical assistance in program design and irrplementation 

Technical assistance in program management and 
administration 

Training in youth development 
Training in academic enrichment and literacy development 

Training in health promotion/education and support services 

Training in w orking w ith a diverse, multicultural student 
population 

Training in classroom management 
Training in athletic instruction 
Training in fine and performing arts activities 
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Percent who reported receiving technical assistance 
or training "to some extent" or "to a great extent" 
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Option I program directors reported moderate levels of satisfaction with 
the training and technical assistance received through the OST initiative. Twenty- 
six percent reported that the training and technical assistance served their 
purposes completely, and another 64 percent reported that it was a good start. 
Thirty-nine percent of program directors reported that implementing the ideas and 
strategies presented in training and technical assistance had improved their 
project, and 43 percent were in the process of implementing the strategies. 

Of the program directors who reported that their program did not 
implement the ideas and strategies presented in training and technical assistance, 
the most common obstacle preventing them from implementing the techniques 
learned was that they needed further training (reported by 50 percent of program 
directors). 

More than half of program directors reported that additional training on 
the following topics would benefit their staff: 



Youth development (75 percent) 

Academic enrichment and literacy development (59 percent) 
Classroom management (57 percent) 
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Fine and performing arts (5 1 percent) 



Program directors most frequently reported that the training topics they 
personally found most useful were; staff supervision (54 percent), program design 
(52 percent), and program management (5 1 percent). 



Opportunities for Career Advancement 

Executive directors of provider organizations reported offering 
opportunities for staff working in OST programs to advance their careers. Such 
opportunities can provide incentives for staff to remain employed by the provider, 
and contribute to improving the quality of OST programming and staff. More 
than three-quarters of executive directors reported offering professional 
development opportunities as a way for program staff to advance their careers, as 
well as opportunities to move to other positions within the organization (90 and 
77 percent, respectively). Somewhat fewer executive directors reported 
opportunities for promotions or raises within the DYCD OST program (67 and 58 
percent, respectively). 



Exhibit 18 

Opportunities for Career Advancement for OST Staff, 
as Reported by Executive Directors (n=158) 



Professional development 

Move to positions w/ ithin org 
outside of DYCD program 

Romotion w ithin the DYCD 
program 

Raises w ithin the DYCD 
program 

0 20 40 60 80 100 

Percent of executive directors who responded "to a great extent" or "somewhat” 




Program Structures and Partnerships 

In the first year of the OST initiative, program directors worked to create 
program policies and procedures that ensured compliance with city and state 
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requirements and that provided the foundation for positive program environments 
in which participants could thrive. 

Program policies. More than three-quarters of Option I program directors 
reported “to a great extent” that their program had policies in place for the 
following purposes: for reporting suspicions of child abuse and neglect, for 
dealing with participant behavior, and for making sure that the time allowed for 
activities was generally appropriate (89, 78, and 78 percents respectively), as 
shown in Exhibit 19. Because of their importance, policies for reporting 
suspicions of child abuse or neglect were a particular priority of DYCD. In 
addition to the 89 percent of program directors who reported that such procedures 
were in place “to a great extent,” another 8 percent said that such procedures were 
in place “to some extent.” Ninety-two percent of elementary-grades program 
directors, 97 percent of middle -grades directors, and 77 percent of high school 
directors reported that these policies were in place “to a great extent” or “to some 
extent.” A total of 99 percent of elementary-grades Option I program directors 
reported that such policies were in place “to a great extent” or “to some extent.” 



Exhibit 19 

Existence of Program Poiicies, 
as Reported by Option I Program Directors (n=392) 



Procedures are in place to report suspicions of child abuse 
and neglect 

Procedures for dealing w ith participant behavior issues are 
in place and effective 

Tfie time allowed for activities is generally appropriate 

This program has links to organizations w here w e can refer 
participants in need of additional services 
Groups are small enough for staff to meet participants' 
individual needs 

Participants make adequate progress w ith homew ork in the 
time provided 

Participants with special needs are successfully integrated 

Participants have freedom in selecting activities 

This program has a process in place for obtaining 
participant input and suggestions 
Staff provide homew ork help to bilingual participants in their 
native language 

Participants have regular opportunities to lead activities 

Participants have regular opportunities to spend time alone if 
needed 




Percent who reported " to a great extent" 
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Program directors were consistent in their reports of challenges to 
implementing high-quality programming. The challenges that they indicated most 
often were that families were not sufficiently involved in their children’s 
participation in the program and that programs did not have sufficient funds to 
provide high-quality programming (64 percent and 52 percent, respectively), as 
shown in Exhibit 20. Program directors also noted low participation rates as a 
challenge; 42 percent said that youth drop out because they lose interest, and 40 
percent said that youth do not attend OST programming regularly enough to have 
enriching experiences. 



Exhibit 20 

Challenges to Implementing High-Quality Programming, 
According to Option I Program Directors (n=392) 



Families are not sufficiently involved in their child/children's 
participation at the program 

We do not have sufficient funds to provide high-quality 
programming 



Youth drop out because they lose interest 



Youth do not attend the OST program regularly enough to 
have enriching experiences 

The school(s) our participants attend would like our 
program to be more academically focused 
The staff at the school(s) participants attend do not 
respond to our requests to coordinate services or 
resources 

We cannot recruit enough youth to participate 



We do not have sufficient administrative support to fulfill 
OST grant reporting requirements 

The space available for our program is inadequate, 
inappropriate, or unsafe 

We have insufficient information about the needs of the 
participating youth in our program 

We have inadequate instructional materials or programming 
ideas 

We do not receive sufficient support or feedback from 
DYCD 
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Percent who reported a "major challenge" or a "minor challenge" 
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Program partnerships with schools. In general. Option I program 
directors reported inconsistent communication with school-day principals, 
teachers, and other key staff. As expected, center-based program directors 
reported significantly less communication with school staff than did directors of 
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school-based programs. Among school-based programs, more than half of 
program directors reported communicating with school staff at least once a month 
on the following topics, as shown in Exhibit 21: 

■ The needs or progress of individual students (70 percent) 

■ Issues related to classrooms/sharing space (63 percent) 

■ Homework assignments (62 percent) 

■ Planning OST program content (61 percent) 

■ Student discipline policies (58 percent) 



Exhibit 21 

Communication with Schooi Staff, as Reported 
by Directors of Option I Schooi-Based Programs (n=270) 



The needs or progress of individual students 
Issues related to classrooms/sharing space 
Homew ork assignments 
Ranning OST program content 
Student discipline policies 
Curriculum concepts currently being taught in school 
OST program enrollment policies 
OST program staffing 
State and local standards in reading, math, and/or science 
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Percent who reported communicating with school staff at least once a month 



Program line staff reported minimal interaetions with sehool-day staff. 
Among the staff who responded to our survey from the 15 programs in the in- 
depth study sample, fewer than a third reported that they eommunieated with 
school staff on a monthly basis about any of the topics we asked about. Among 
those engaging in sueh eommunieation, the most frequent topic of monthly 
conversations with sehool-day staff was about homework assignments (reported 
by 32 pereent of staff). 

Some sehools and programs implemented formal structures for 
eommunieation. For example, in one sehool, all organizations providing after- 
school services met two times a month with the prineipal. The meeting provided 
eaeh organization with an opportunity to diseuss issues related to the use of spaee 
and the eoordination of their various after-sehool aetivities. “We give updates 
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and talk about issues that have eome up.” In another program the kindergarten, 
first-, and seeond-grade teaehers gave program staff a log of homework for eaeh 
week, “so we know exaetly what they have to do eaeh week” and ean use the 
program time effectively. In many other programs, staff communicated 
informally through conversations with teachers about student behavior and 
academic progress. 

Program partnerships with other organizations. Three-quarters of 
program directors (75 percent) reported that outside organizations, in addition to 
the provider, offered special programs, activities, or services for youth at their 
program. More than half of programs reported that outside organizations donated 
materials or supplies (58 percent), provided funding through grants or contracts 
(57 percent), referred students to the OST program (56 percent), and provided 
special programs, activities, or services to families (56 percent). 



6. Implementation of Process and Content 
Features in Option I Programs 

As suggested by the theory of change for the evaluation of OST programs 
(Exhibit 2), the implementation of activities that include rich program content, 
varied content delivery strategies, and efforts to foster positive relationships is 
essential to the success of out-of-school time programs in achieving positive 
short- and long-term outcomes in participants. 



Program Content 

The DYCD OST initiative encouraged programs to offer a variety of rich 
content-based activities designed to support participants’ academic and social 
development, consistent with the developmental needs of the youth served. 

The DYCD REP required all Option I elementary-grades programs to 
provide opportunities and support for participants to complete their homework; in 
addition, all elementary- and middle-grades programs were encouraged to provide 
other academic enrichment activities that augmented school-day instruction. 
Elementary-grades programs were also asked to offer participants a balanced mix 
of activities that included recreation and physical activity. Middle-grades 
programs were required to include activities that addressed the transition from 
elementary to middle school, including opportunities for positive identity 
formation, leadership development, and learning related to health and sexuality. 

In contrast, high school programs were not required to have an academic focus 
but were required to offer project-based programming that focused on topics such 
as careers, job training and placement, preparation for post-secondary education, 
and life skills/transition to independent living. Programs in Options II and III that 
served multiple grade levels were required to provide activities that promoted 
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positive relationships and high expeetations and that supported the exploration of 
interests and development of skills and creativity. 

Elementary-grades program activities. The program director survey 
asked directors to identify activities that were ever offered in their program, 
offered to most participants, and offered at various levels of frequency. For 
analyses of Year 1 program implementation, evaluators focused on the activities 
that program directors reported offering with the most regularity: activities that 
were offered to most or all participants throughout the year for at least one hour 
per week. 

Exhibit 22 presents the 10 activities that program directors most frequently 
reported offering for at least an hour per week. Nearly all program directors 
consistently offered homework help to their elementary-grades participants (98 
percent). In addition, 84 percent reported offering visual arts and crafts activities. 
More than half of programs also regularly offered free time for physical play and 
unstructured time for socializing (59 percent and 57 percent, respectively). Other 
activities reported to be regularly offered by at least half of elementary-grades 
program directors included dance/movement activities (57 percent) and organized 
team sports (50 percent). 



Exhibit 22 

Ten Most Common Eiementary-Grades Program Activities, 
as Reported by Option 1 Program Directors (n=145) 



Homew ork help )8 

Visual arts and crafts instruction/ projects 
Organized reading activities 
Learning games/activities 
Group instruction in specific academic subjects 
Recreational reading 
Math games/activities 
Organized w riting activities 
Free time for physicai play /pick-up sports 
Unstructured time for socializing 
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In interviews, elementary-grades program directors typically highlighted 
program activities that supported DYCD OST initiative goals. For example an 
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elementary-grades program director emphasized the program’s goal to strengthen 
participants’ academic skills. “For academics we’re interested in mastery. We 
use Voyager, and we have [certified teachers] on our staff whose role is to work 
with our group leaders in implementing our curriculum. . .we have a math coach 
who is on staff so the kids get literacy and math three times a week for 45 minutes 
at a time, and they also get 45 minutes of homework help [everyday except 
Friday]...” 

Middle-grades program activities. Exhibit 23 displays the 10 activities 
most frequently offered to middle-grades participants at least an hour a week, and 
shows a similar list to that of elementary-grades programs. In comparison, 
organized team sports were more common in the middle-grades programs 
(reported by 73 percent of program directors). In addition, slightly more than half 
(5 1 percent) of middle-grades program directors reported regularly offering 
unstructured time for socializing. 



Exhibit 23 

Ten Most Common Middie-Grades Program Activities, 
as Reported by Option I Program Directors (n=117) 
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Organized team sports instruction/ activities 
Visual arts and crafts instruction/ projects 
Free time for physical play /pick-up sports 
Learning games/activities 
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Dance/movement instruction or activities 
Math games/activities 
Recreational reading 
Organized reading activities 
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High school program activities. High school program directors reported 
the widest range of consistently offered program activities. In fact, as shown in 
Exhibit 24, only one activity — ^unstructured time for socializing — ^was reported by 
more than half of the program directors as being regularly offered to all 
participants (reported by 56 percent), suggesting that high school OST programs 
were more specialized and that each targeted a smaller set of program activities. 
These activities tended to be more civic-oriented than in the elementary- and 
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middle-grades programs. For example, while 40 percent of directors reported 
regularly providing homework help to most of their high school youth, 49 percent 
had frequent peer discussions of topics important to youth, and 44 percent 
reported frequent discussions of current events. 

One high school program director emphasized the program’s efforts to 
develop these communication and expression skills among participants. “I think 
at [the high school] level they’re venturing into a stage of independence, and we 
want to enhance that freedom of expression and develop an environment that they 
belong to instead of us dictating. When you [dictate to students] you lose them to 
the street. We learned that we should listen more to what they want. . .it’s youth 
development and leadership, it engages them and gives them input into what they 
want.” 



Exhibit 24 

Ten Most Common High Schooi Program Activities, 
as Reported by Option 1 Program Directors (n=115) 
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Discussion of current events 
Career exploration activities 
Mentoring 
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Content Delivery Strategies 

Staff instructional strategies. Although OST programs strived to 
implement a broad range of activities during the first year of the initiative, they 
sometimes struggled in implementing the quality features associated with positive 
student outcomes in OST programming. High-quality OST programs typically 
promote mastery through activities and strategies that provide their participants 
with both structured and unstructured learning opportunities and that promote 
participant autonomy, choice, and leadership (Bransford, Brown, & Cocking, 
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1999). Through program observations, evaluators recorded and assessed the 
extent to which staff instructional strategies fostered mastery, as measured by the 
following indicators (see Appendix C for reliability data on the evaluation’s 
observation-based scales): 

■ Youth contribute opinions, ideas, and/or concerns to discussions 

■ Staff encourage youth to share ideas, opinions, and concerns 

■ Staff communicate goals, purposes, and expectations 

■ Staff verbally recognize youth’s efforts and accomplishments 

■ Staff ask youth to expand on their answers and ideas 

■ Staff challenge youth to move beyond their current level of 
competency 

■ Staff employ varied teaching strategies 

In visits to programs in the in-depth study sample, observers rated each of 
these items on a scale of one to seven, and items were averaged in analysis to 
compute a scale measuring mastery/skill-building instruction. Across all 
activities observed, the average scale score was 3.06 out of 7. In years 2 and 3 of 
the evaluation, observation ratings will be compared against the scale scores 
reported here to gauge changes in program quality. In Year 1, notable differences 
were evident in the extent to which staff delivered mastery-oriented instruction 
across activity types and across grade levels served. 

■ Homework help and tutoring activities scored significantly lower 
on the mastery scale compared to all other activity types (average 
score of 2.68, compared to 3.26). 

■ In contrast, academic enrichment activities were rated significantly 
higher than were other activity types (3.62, compared to 2.87). 

■ Activities in which certified teachers or activity specialists were 
present were rated significantly higher than other activities (3.72, 
compared to 2.92) in terms of the extent to which instruction 
focused on skill-building and mastery. 

■ Activities that were intentionally designed to promote skill- 
building were, as might be expected, rated significantly higher than 
were activities designed to allow students to practice existing skills 
or complete homework (4.07, compared to 3.32 and 2.55, 
respectively). These activities included, for example, targeted 
vocabulary-building sessions with ELL students in a middle-grades 
program and a dance class in a high school program where 
students learned the choreography of a new dance. 
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■ Activities in high school programs were rated significantly higher 
than were activities in elementary- or middle-grades programs 
(4.24, compared with 2.90 and 2.73, respectively). 

Quality of activity content and structure. In activity observations, 
evaluators also rated the quality of the activity content and structure on a scale of 
one to seven, as measured by the following items: 

■ Activity is well organized 

■ Activity challenges students intellectually, creatively, 

developmentally, and/or physically 

■ Activity involves the practice/progression of skills 

■ Activity requires analytic thinking 

Because of the nature of homework help activities, in which programs had 
little control over the content and skills targeted by the homework, these were 
excluded from analysis. Among the remaining activities observed, the average 
rating on the activity content and structure scale was 4.26 out of 7, but with 
significant differences noted in the quality of activity content: 

■ Arts activities scored significantly higher on the activity content 
and structure scale than did other activities (4.88, compared to 
4.00). 

■ Not surprisingly, intentionally open and unstructured activities 
were rated significantly lower in terms of the quality of activity 
content and structure than were other activities (2.98, compared to 
4.46). 

■ Activities in which certified teachers or specialists were present 
scored significantly higher on the activity content and structure 
scale (4.93, compared to 4.13) than were other activities. 



Positive Reiationships 

High-quality OST programs seek to foster the development of positive 
staff-youth relationships as well as positive peer relationships among youth. 
Through activity observations, evaluators rated the extent to which activities 
displayed positive relationships based on the following indicators, each rated on a 
scale of one to seven. Evaluators then combined these items into scales with an 
average score of 5.31 out of 7, suggesting generally positive relationships across 
all activities observed. 

■ Youth are friendly and relaxed with one another 

■ Youth respect one another 
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■ Youth show positive affect to staff 

■ Youth are on-task 

■ Youth listen actively and attentively to peers and staff 

■ Staff use positive behavior management techniques 

■ Staff are equitable and inclusive 

■ Staff show positive affect toward youth 

■ Staff attentively listen to and/or observe youth 

There were few significant differences in the extent to which evaluators 
observed a positive atmosphere in the activities. Most notably, homework and 
tutoring activities were associated with a significantly lower atmosphere rating, 
compared to other activity types (4.97, compared to 5.49). 



7. Program Implementation in Option II and 
Option III OST Programs 

In addition to the Option I programs described above, in Year 1 of the 
OST initiative, DYCD also launched Option II and Option III OST programs. 
Option II programs were designed to build on public-private partnerships and 
were required to receive at least 30 percent of their funding from private sources 
such as corporations, foundations, and individuals. Option III programs were 
operated through the Department of Parks and Recreation and were to be offered 
at Parks sites. Because of the different structures and expectations of Option II 
and Option III programs, evaluators analyzed the program director survey data 
separately and report here on program implementation in Year 1 for Options II 
and III. 



Participant Recruitment 

Option II and III programs sought to serve students aged 6 to 21 across all 
grade levels in all ten DOE regions. One set of Option III programs specifically 
sought to serve students with disabilities and provided academic support so that 
students could meet or exceed academic standards. 

Sixty-five percent of Option II program directors reported that they 
offered open enrollment for all interested youth. Forty-three percent of Option II 
programs also reported that they sought to serve youth who were recommended 
by school-day teachers or counselors. 

All 10 responding Option III program directors reported that they allowed 
open enrollment for all interested youth. Five of 10 Option III programs reported 
trying to serve youth who scored below proficient on their city or state 
assessments, and five of 10 programs reported trying to serve youth identified by 
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their school as needing special assistance in reading and/or math, reflecting the 
intent for Option III programs to include an academic support component. 



Objectives of Option II and III Programs 

Option II program directors overwhelmingly reported that the major 
objective of their program was to provide a safe environment for youth (96 
percent). Three-quarters or more of Option II program directors also reported that 
a major objective of their program was to: 

■ Help youth develop socially (95 percent) 

■ Provide youth with positive adult guidance and/or mentors (91 
percent) 

■ Provide health, well-being, and life skills (80 percent) 

■ Promote respect for diversity among youth (76 percent) 

■ Provide leadership opportunities for youth (75 percent) 

Option III programs also reported having a wide array of objectives. All 
10 Option III programs reported that providing a safe environment for youth was 
a major objective. Seven or more Option III program directors also reported that 
a major objective of their program was to: 

■ Provide youth with positive adult guidance and/or mentors (8 
programs) 

■ Promote respect for diversity among youth (7 programs) 

■ Help youth improve their academic performance (7 programs) 

■ Help youth develop socially (7 programs) 



Activities Offered in Option II and Option III Programs 

Because Options II and III offered flexibility in scheduling OST 
programming, the evaluation analyzed the activities that program directors 
reported offering to all or most youth in every month in which the program was 
open, with no restriction on the frequency of youth participation. Option II 
program directors most frequently reported offering discussion-based activities, as 
shown in Exhibit 25. More than half of Option II program directors reported 
offering the following discussion activities to the majority of participants on an 
ongoing basis: 



■ Peer discussion of topics that are important to youth 

■ Discussion of diversity issues 

■ Mentoring 

■ Discussion of issues, events, or problems in the community 

■ Discussion of current events 
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Exhibit 25 

Option II Program Activities, as Reported by Program Directors 

(n=79) 
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Peer discussion of topics that are inportant to youth 
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Discussion of issues, events, or problems in your 
community 

Mentoring 

Discussion about diversity issues 
Organized w riting activities 
Discussion of current events 
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Option III program director reports of activity offerings demonstrate a 
focus on academic enrichment activities, as shown in Exhibit 26. At least seven 
of the 10 responding program directors reported regularly offering the following 
activities: 



■ Peer discussion of topics important to youth 

■ Organized writing activities 

■ Organized reading activities 

■ Recreational reading 
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Exhibit 26 

Option Ilf Program Activities, as Reported by Program Directors 

(n=10) 

Peer discussion of topics that are inportant to youth 
Organized w riting activities 
Recreationai reading 
Organized reading activities 
Discussion about diversity issues 
Homew ork heip 
Discussion of current events 
instruction in iife skiiis 
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Program Director Quaiifications and Supports 

Option II program directors typically had prior experience as camp 
counselors/leaders or as youth workers (69 percent and 62 percent, respectively). 
Eighty-six percent of Option II program directors reported that they had 
completed at least a four-year college degree, and 5 1 percent had completed a 
master’s degree or higher. In addition, 22 percent of Option II program directors 
were certified to teach. Fifty-nine percent of Option II program directors reported 
being white, 20 percent were African-American, and 16 percent Hispanic/Latino. 
The majority of Option II program directors (76 percent) were female. 

Six of 10 Option III program directors reported prior experience as a camp 
counselor/leader, and six had experience as an administrator in a social services 
organization. Reflecting the partnership with the Department of Parks and 
Recreation, five Option III program directors reported prior experience as an 
administrator at a child/youth center or at a park or recreation center and as a 
recreation, youth, or child care worker. Six of ten Option III program directors 
had completed a four-year college degree or higher, and the remaining four 
completed some college or other classes/training not related to a degree. Only 
one Option III program director reported being certified to teach. Four of ten 
Option III program directors were Hispanic/Fatino, three were African-American, 
and three were White. Half of the ten Option III program directors were female. 
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Program Staff Qualifications and Supports 



Option II OST programs employed relatively few staff members. Nearly 
three-quarters reported having between one and ten paid staff members (73 
pereent). Nearly half (47 pereent) of Option II programs had between one and 10 
volunteer staff members; more than a third (38 pereent) had no volunteers. 

Similar to Option I programs, Option II programs relied on a eombination of 
experienced professionals and young staff members. Eighty-nine percent of 
Option II programs had staff with college degrees or higher, and 32 percent of 
Option II programs employed some staff with teaching certificates. About a 
quarter of Option II programs (26 percent) employed teen staff members. More 
than half (53 percent) of Option II programs reported that the average hourly 
wage for staff members was less than $16 per hour. Most Option II programs (64 
percent) did not have a master teacher or education specialist; 20 percent of 
Option II programs had one such staff member in a paid part-time position, and 
1 5 percent had one in a paid full-time position. 

All Option II programs indicated that finding qualified staff to hire was a 
challenge. Most Option II programs also reported that they were challenged to 
offer the competitive salaries necessary to hire qualified staff (70 percent), that 
they could not find volunteers with the time and expertise needed to help in the 
program (64 percent), and they could not afford to offer potential staff enough 
hours of paid employment (58 percent). 

Eight of 10 Option III programs reported having between one and ten paid 
staff members. Nine Option III programs reported having no volunteer staff 
members. Eight Option III programs reported employing some staff with a 
college degree, seven programs reported having teen staff, and six programs 
reported having staff with teaching certificates. Staff wages in Option III 
programs were relatively high. Average hourly wages for staff were widely 
distributed, with the highest number of programs (four) reporting average hourly 
wages of $26 to $30.99 per hour. Nonetheless, all 10 Option III programs 
reported that finding qualified staff to hire was a challenge. 



Technical Assistance and Supervision 

All Option II program directors reported holding staff meetings at least 
once a month, and 41 percent reported holding staff meetings at least once a 
week. Sixty-eight percent of Option II program directors reported that they 
require at least some staff to submit activity or lesson plans on a regular basis. A 
little less than half of Option II programs (46 percent) reported using published or 
externally developed curriculum to guide their activities. Eighty-two percent of 
program directors reported that they and their staff received technical assistance 
using OST Online “to some extent” or “to a great extent.” This was the only type 
of training for which more than half of Option II program directors reported that 
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they and their staff reeeived technieal assistanee or training. Seventy-one pereent 
of Option II program direetors reported that the type of training that would be 
most helpful to their staff was in the area of youth development. Fifty-six pereent 
of Option II program direetors reported that the type of training that would be 
most helpful to them was in the area of staff supervision. 

All 10 Option III program direetors reported holding at least monthly 
meetings with their OST program staff, with seven program directors holding 
them at least once a week. Eight of 10 Option III program directors reported that 
they did not ask staff to submit activity or lesson plans. Six Option III program 
directors reported requiring all or most staff to submit activity or lesson plans on a 
regular basis. All 10 Option III program directors reported receiving technical 
assistance in using OST Online, and five Option III program directors reported 
receiving technical assistance in program design and implementation. Six Option 
III program directors indicated that the training topic that would be most useful to 
their staff was academic enrichment and literacy development, six reported that it 
was youth development and five reported that it was training in OST Online. 



8. Participant Engagement in OST Services 

Program attendance is an essential indicator of program engagement with 
its target population and of effectiveness. Evidence from prior evaluations makes 
clear that regular program attendance is strongly associated with the development 
of the types of positive youth outcomes sought through the OST initiative (Huang, 
Gribbons, Kim, Eee, & Baker, 2000; Reisner, White, Russell, & Birmingham, 
2004). DYCD monitors program attendance using OST Online and, in particular, 
using the system’s template for program reporting of individual youth 
participation in specific activities to which the youth has been assigned. 

The DYCD initiative established clear standards for program participation 
that vary based on program option and grade level served. Eor contractual 
purposes, DYCD monitored program-level participation rates according to the 
following standards: 

■ The average daily attendance for an Option I elementary-grades 
program was expected to be 80 percent of the program’s target 
enrollment. These programs were expected to serve youth for a 
minimum of three hours a day, five days a week, for 36 weeks, 
plus 20 days of OST services 10 hours per day during school 
vacations, for a total of 740 hours. 

■ Option I middle-grades programs were required to offer 
programming for at least eight hours per week for 36 weeks, or 
488 hours of service per contracted participant, including 20 days 
of OST services 10 hours per day during school vacations. These 
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programs were expected to provide 75 percent of the expected 
number of hours of service during the contract year, based on their 
contracted enrollment number. 

■ Option I high school programs were expected to provide a 
minimum of 108 hours of service per year per participant. These 
programs were expected to provide 70 percent of the expected 
number of hours of service during the contract year, based on their 
contracted enrollment number. 

■ Option II programs were expected to provide a minimum of 160 
hours of service per year per participant. These programs were 
expected to provide 70 percent of the expected number of hours of 
service during the contract year, based on their contracted 
enrollment number. 

The evaluation examined program attendance through questions posed in the 
participant survey and through analysis of participation data in OST Online. 



Participation in OST Programs 

Analyses of data collected through OST Online permitted evaluators to 
quantify program attendance patterns. From the OST Online record of 
participants’ attendance in their assigned activities, it is possible to determine 
each participant’s number of days or hours of program attendance during the 
program year. For analyses of Year 1 participation patterns, evaluators focused 
on programs that provided services for the full school year and that also entered 
reliable activity participation data into OST Online.^ Analyses presented here are 
based on data entered by 41 1 OST programs during the period September 2005 
through June 2006. 

Experience in the first year of OST operations indicated that tracking and 
entering daily attendance by activity for each youth was labor-intensive and hence 
could have resulted in inaccuracies. For example, after-school programs do not 
always operate according to schedule, due to unexpected events and opportunities 
and due also to staff absences. Programs know to expect this and adjust their 
daily activities accordingly, but OST Online assumes a consistent schedule of 
activities as planned at the beginning of the program year or semester. Therefore, 
a code of “not present” for a participant on a given day does not distinguish 
between a participant absence and an activity cancellation. Because of problems 



^ For analysis purposes, the evaluation exeluded programs that operated for less than nine months 
and programs with partieipation data-entry errors in 40 pereent or more of the student reeords 
within that program. Individual students from the remaining programs who attended their OST 
program for four days or fewer and those with partieipation data-entry errors were also exeluded 
from analysis. 
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such as these in the reeording of aetivity data, evaluators foeused on the numbers 
and pereents of youth present in the program on a given day and not on aetivity- 
speeifie attendanee. 

Based on the standards for program partieipation of OST programming 
established by DYCD, the evaluators ealeulated the minimum number of hours 
eaeh Option I partieipant was expeeted to reeeive during Year 1, as shown in 
Exhibit 11 In the elementary- and middle-grades, partieipants did not attend as 
many hours, on average, as they were expeeted to attend. Partieipants in 
elementary-grade programs attended their OST program for an average of 3 1 1 
hours (72 pereent) during the program year and were expeeted to attend for an 
average of 432 hours. Partieipants in middle-grades programs also fell short of 
their targeted attendanee, by attending for an average of 130 hours (60 pereent) in 
Priority Middle Sehool programs and an average of 154 hours (74 pereent) in 
other middle-grades programs, eompared to the target of 216 hours. Option I high 
sehool partieipants exeeeded the targeted number of hours. High sehool 
partieipants attended for an average of 97 hours (128 pereent), eompared to the 
targeted attendanee of 76 hours. On an individual level, a quarter of elementary- 
and middle-grades partieipants met the targeted number of hours (25 pereent and 
26 pereent, respeetively), while 39 pereent of high sehool partieipants did so. 



Exhibit 27 

OST Participants’ Actuai and Targeted Mean Attendance 

(in hours) 
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Exhibit reads: Youth in eiementary-grades Option I programs were expected to attend 
their OST program for an average of 432 hours and actuaily attended for an average 
of 31 1 hours during the measurement period. 



At the elementary level, programs are expected to offer programming for a minimum of three 
hours a day, five days a week, for 36 weeks, plus 10 hours a day over 20 vacation days for a total 
of 740 hours. For purposes of computing the expected number of hours of participation, 
evaluators used the daily service-availability guidelines but excluded the OST service hours 
expected on school-closing days, which produced a total of 540 hours. Adapting DYCD’s 
program-level rate of participation requirement, the evaluation set the expected number of hours 
for an elementary-grade participant at 80 percent of this level, or 432 hours. Using similar 
calculations, the expected number of program hours for middle-grades and high school 
participants were set at 216 and 76 hours, respectively. All students were measured against these 
standards regardless of how long they were enrolled in the program. The evaluation did not have 
enough information on youths’ individual enrollment periods to adjust the standard for students 
who were not enrolled for the full program period. 
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OST Online captures the specific activities and schedule to which an 
individual participant was assigned. Each OST participant may therefore have a 
unique standard of expected program attendance. For example, while most 
elementary-grades participants may be assigned to one or more daily program 
activities five days per week, a high school participant might be expected to 
attend an activity on only one day a week. For each day, program staff record in 
OST Online whether the youth attends each activity to which he or she is 
assigned. Evaluators received reports of problems and delays in entering 
program-level data into OST Online but could not estimate the extent of these 
problems. 

Fevels of participation can also be expressed as an attendance rate, which 
reflects mean attendance levels of enrolled youth, as shown in Exhibit 28. Youth 
attending elementary-grades programs in Option I attended the program on 62 
percent of the days that they were assigned to an activity. Attendance rates were 
lower among programs serving older youth. Y outh attending middle-grades 
programs in Option I attended their program on 46 percent of the days they were 
assigned to an activity (45 percent for youth in Priority Middle School programs), 
compared with 40 percent for Option I high school participants. Youth enrolled 
in Option II programs attended their programs on 52 percent of the days they were 
assigned to an activity. In Option III programs, youth attended 42 percent of their 
assigned days. 



Exhibit 28 

Average Attendance Rates in Assigned Activities (in percents) 
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Other Middle 
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All 
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45 


46 


40 
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Exhibit reads: Youth enroiied in eiementary-grades Option I programs attended 62 percent of the 
days they were assigned to an activity. 



As would be expected from the preceding data, elementary-grades 
participants were less likely to report in the participant survey that they were 
unsupervised during the after-school hours than were middle-grades or high 
school participants. Eighty-five percent of participants in elementary-grades 
programs said that they never went to a place after school where adults were not 
present, compared to 77 percent of middle-grades participants and 56 percent of 
high school participants responding in the same way. These findings are 
important in light of research demonstrating the very poor youth outcomes 
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associated with an absence of after-sehool supervision (Mahoney & Stattin, 2000; 
Vandell, Pierce, Brown, Lee, Bolt, Dadisman, Peehman, & Reisner, 2006). 

Youth, partieularly high sehool students, also reported attending non-OST 
programs or after-school activities on oecasion. Sixty percent of youth who 
participated in high school OST programs also reported participating in a different 
after-school activity at least onee a week, compared to 38 pereent of elementary- 
grades youth and 37 percent of middle-grades participants. OST participants also 
reported spending after-sehool time in a home, supervised by an adult, at least 
once a week (61 pereent of elementary participants, 70 percent of middle-grades 
partieipants, and 75 percent of high school participants). In addition, 57 percent 
of high sehool partieipants and 21 pereent of middle-grades partieipants reported 
going to an after-school job at least one afternoon a week. 



Attendance of Participants in Target Zip Codes 

As seen in Exhibit 29, youth enrolled in programs in target zip codes 
attended their program at the same rates or higher as youth attending programs in 
non-target zip eodes, indicating the needs in these neighborhoods for OST 
services and their reeeptivity to the programs. Youth in middle-grades and high 
sehool programs in the two types of communities attended programs at similar 
rates. In programs serving the elementary-grades and multiple grade levels, youth 
in programs loeated in the target zip codes attended their programs at higher rates 
than their non- target zip code counterparts. Elementary-grades youth in target zip 
eodes attended their programs on 63 percent of the days they were assigned, 
compared to 62 pereent for students in non-target zip eodes. In programs serving 
multiple grade levels, youth in target zip codes attended their programs on 62 
pereent of the days they were assigned, compared to 46 percent for students in 
non-target zip codes. 
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Exhibit 29 

Assigned Program Days Attended, 
by Grade Levei and Target Zip Code (in percents) 




Exhibit reads: Students enroiied in eiementary-grades programs in target zip codes attended 
their program on 63 percent of the days they were assigned to an activity. 

* p<.05, **p<.01, ***p<.001 



Attendance in Center-Based and Schooi-Based Programs 

In general, youth enrolled in center-based programs attended at slightly 
higher rates than youth in school-based programs, as shown in Exhibit 30. This 
was true at elementary-grades, middle-grades, and high school levels, where 
attendance rates of 63 percent and 62 percent were computed at the elementary 
level, 49 percent and 46 percent at the middle-grades level, and 41 percent and 39 
percent at the high school level, for center-based and school-based programs 
respectively. At programs serving multiple grade levels, youth at school-based 
programs attended their programs at a higher rate than students at center-based 
programs, at 59 percent and 47 percent, respectively. 
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Exhibit 30 

Assigned Program Days Attended, 
by Grade Levei and Program Location (in percents) 
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Exhibit reads: Students enrolled in elementary-grades center-based programs attended their 
program on 63 percent of the days they were assigned to an activity. 

*p<.05, **p<.01, ***p<.001 



9. Social Development Outcomes of Youth 

Out-of-school time programs play an important role in helping youth to 
develop the soeial skills with both adults and peers that they need in order to 
mature into successful adults themselves. With the current high-stakes demands 
of the school day, students may have few opportunities to form and sustain 
friendly, collaborative relationships with other peers or adults. Compared to the 
school day, OST opportunities plaee much higher priority on the role of 
relationships, and they typically encourage many forms of positive adult and peer 
friendships and collaboration. The evaluation measured social development 
through partieipant-survey items addressing interactions with peers and adults and 
also through items measuring the development of leadership skills. As noted 
earlier, the participant surveys were administered to youth in grades 3-12 in a 
random sample of OST programs. See Appendix D for teehnieal details regarding 
the evaluation’s analyses of partieipant experiences. 



Exposure to New Experiences 



By providing engaging programming that exposes youth to opportunities 
that they would not otherwise have, OST programs can increase their draw for 



51 



participants and their capacity to contribute to youths’ academic and social 
development. In interviews, participants cited exposure to new experiences and 
new activities as primary reasons why the program engaged them. For example, 
one participant enjoyed attending the program “because you get to learn a lot of 
new things in after school. [You learn] how to do something [and] to know how 
to do it.” Another participant described an activity in which the youth learned 
about the earthquake that occurred in Pakistan and planned a fundraiser to help 
the victims. “We raised enough money to send over there, so they could get tents 
and food.” 

Overall, about half of OST participants strongly agreed that their program 
had exposed them to new and interesting activities in Year 1. Across all grade 
levels, 53 percent of youth agreed a lot that the program gave them a chance to do 
a lot of new things. In addition, more than half reported that the program 
activities really got them interested (56 percent). 

Participant reports of engaging in new experiences through the OST 
program varied significantly by grade level, as shown in Exhibit 3 1 . Elementary- 
grades youth were most likely to report that the OST program activities engaged 
them in the program. Significantly more also agreed that they worked on projects 
that really made them think than did participants in either middle-grades or high 
school programs (46 percent, compared with 34 percent and 36 percent, 
respectively). In addition, elementary-grades students were significantly more 
likely to report that the activities really got them interested than were high school 
participants (59 percent, compared with 51 percent). 

However, as seen in Exhibit 31, participants in high school programs were 
significantly more likely than their peers in middle-grades programs to agree a lot 
that they were able to do things that they usually don’t get to do anywhere else 
(46 percent, compared with 39 percent). High school students reported this 
exposure to new experiences despite more limited OST activity choices than the 
younger students, with only 42 percent reporting that there was a lot for them to 
choose to do in the program, compared to 48 percent of elementary-grades 
participants and 52 percent of middle-grades participants. 
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Exhibit 31 

Participant Reports of Exposure to New Experiences, by Grade Levei 



In this program... 



The activities realiy get me interested'' 



I get a chance to do a lot of new things 



There is a iot for me to choose to do*" 



I get to w ork on projects that reaily make me think"' 



I get to do things that I don't usually get to do anyw here else" 
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Percent who "agreed a lot" 

■ High Schooi participants (n=752) □ Middle-grades participants (n=f ,029) □ Bementary-grades participants (n=1 ,80f ) 



* Elementary-grades programs differ significantly (p < .05) from middle-grades programs. 
° Middle-grades programs differ significantly (p < .05) from high school programs. 

'' Elementary-grades programs differ significantly (p < .05) from high school programs. 



Youth in center-based programs reported signifieantly different levels of 
exposure to new experienees through the OST program aetivities than did youth 
attending sehool-based programs. For example, signifieantly more sehool-based 
youth agreed a lot that there was a lot for them to ehoose to do at the OST 
program than did youth in eenter-based programs (5 1 pereent, eompared to 40 
pereent). Similarly, 59 pereent of partieipants in sehool-based programs reported 
that the aetivities in the program really got them interested, eompared to 49 
pereent of partieipants in eenter-based programs. However, youth in eenter-based 
programs were more likely to report that in the OST program they got to work on 
projeets that really made them think, eompared to youth in sehool-based programs 
(43 pereent, eompared to 40 pereent). 



Relationships with Peers 

DYCD and evaluators took a serious interest in the soeial interaetions in 
OST programs beeause of the large body of researeh suggesting the importanee of 
positive soeial relationships in fostering the development of other positive 
personal traits (e.g., Eeeles & Gootman, 2002; Pieree, Hamm, & Vandell, 1999). 



Peer interactions. The partieipant survey asked youth in elementary- 
grades programs a series of questions about their interaetions with their peers, as 
shown in Exhibit 32. Youth most frequently reported that they had a good time 
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playing with other kids in the program, with 70 pereent “agreeing a lot.” 
Partieipants also tended to report that they had a lot of friends in the program (69 
pereent) and that they got to know other kids really well in the program (64 
pereent). These survey data refleeted partieipant eomments during site visits. For 
example, partieipants reported that: 

■ “I have made new friends at the program.” 

■ “I have lots of friends in the program. I am usually quiet in sehool; 
it is different here in after sehool.” 

■ “It’s fun to be able to play with all the kids and go to the gym.” 

■ They just enjoy “being with the other kids.” 



Exhibit 32 

Eiementary-Grade Participant Reports of 
Interactions with Peers (n=1,799) 



In this program I... 

Have a good time playing 
w ith other kids 

Have a lot of friends 

Get to know other kids 
really w ell 

Get along w ith other kids 

Like the other kids 

Can really trust the other 
kids 

0 20 40 60 80 too 

Percent who "agreed a lot" 




For the most part, elementary-grades youth in center-based programs 
reported similar levels of positive interactions with peers as did youth in school- 
based programs, with one exception: youth at school-based programs were 
significantly more likely to report that that they had a lot of friends in the OST 
program, compared to youth in center-based programs (70 percent, compared to 
65 percent). 

Prosocial behaviors. The OST initiative sought to foster positive 
relationships among youth. As such, many programs incorporated conflict- 
resolution curricula to train and encourage youth to solve problems appropriately. 
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The survey asked middle-grades and high school participants questions about 
their prosocial behavior over the past 30 days. The overwhelming majority of 
participants (87 percent) reported cooperating with others in completing a task 
and giving someone a compliment at least once in the past 30 days. Eighty-five 
percent of youth reported that they helped other students solve a problem. In 
addition, more than half of participants reported that they had done the following 
at least once in the past 30 days: 

■ Told other students how they felt when they did something they 
liked (75 percent) 

■ Told other students how they felt when someone upset them (71 
percent) 

■ Helped someone stay out of a fight (63 percent) 

■ Protected someone from a bully (57 percent) 

Responses varied significantly by grade level served but not by program 
location. Eighty percent of youth in high school programs reported that they told 
other students how they felt when they did something they liked at least once in 
the last month, compared to 71 percent of youth in middle-grades programs. 
Seventy-six percent of high school participants reported that they told other 
students how they felt when they upset them, compared to 67 percent of middle- 
grades participants. Eifty-nine percent of middle-grades youth reported that at 
least one time in the last 30 days they protected someone from a bully, compared 
to 53 percent of high school youth. 

Sense of belonging. Overall, participants across all grade levels reported 
relatively high levels of attachment to their OST program. Across all participants, 
74 percent of youth “agreed a lot” that they felt safe in the OST program, and 60 
percent felt like they belonged in the program and that the program was a good 
place to hang out. 

Youth varied in their responses to certain items measuring their sense of 
belonging in the program. Middle-grades participants were significantly less 
likely to feel like their ideas counted in the program than were high school 
participants (44 percent, compared with 50 percent). These middle-grades 
students were also less likely than their elementary-grades peers to report feeling 
safe in the OST program (71 percent, compared with 76 percent). 

Participants in center-based programs were significantly more likely to 
report that they felt a strong attachment to their OST program, as illustrated in 
Exhibit 33. Eor example, 78 percent of youth in center-based programs reported 
that they felt safe in the program, compared to 73 percent of youth in school- 
based programs. Sixty-four percent of youth in center-based programs reported 



55 




that they felt like they belonged, compared to 59 percent of youth in school-based 
programs. Similarly, youth in center-based programs were more likely to feel like 
they matter in the program, compared to youth in school-based programs (60 
percent, compared to 55 percent). Finally, 51 percent of participants in center- 
based programs felt like their ideas counted in the program, compared to 45 
percent of participants in school-based programs. 



Exhibit 33 

Participant Reports of Sense of Beionging, by Program Location 



In this program 1 feel like... 






1 am safe* 
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p < .05 



These differences were accentuated for the high school participants. In 
particular, high school youth in center-based programs were significantly more 
likely to feel connected to the program than were high school participants in 
center-based programs. Most notably, 67 percent of high school participants in 
center-based programs felt like they mattered in the program, compared to only 
51 percent of school-based participants. Similarly, 79 percent of center-based 
high school participants felt safe in the program, compared with 70 percent of 
school-based youth. Significantly more high school youth in center-based 
programs also reported feeling like they belonged (65 percent, compared to 56 
percent in school-based programs). Less than half (47 percent) of school-based 
high school participants felt like their ideas counted, compared to 54 percent of 
center-based participants. 
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Relationships with Adults 



In general, partieipants reported positive interaetions with OST program 
staff members. Across all grade levels, 68 percent of participants “agreed a lot” 
that staff treated them with respect, and 67 percent reported that staff thought that 
they could learn new things, although only 44 percent of youth felt that staff 
always keep their promises. 

In interviews, youth spoke of positive interactions with staff and felt 
comfortable discussing their problems with staff members. “I could go to almost 
every counselor, but not every counselor would know how to solve all the 
problems right then and there, so I would go to them all and get their ideas,” said 
one participant. Another noted, “I talk to [the program director] most of the time 
about things going on; about school and not school things.” In fact, providing 
participants with a place where there were adults “they could trust” was a goal for 
many programs. 

Participant reactions to questions about OST program staff varied by grade 
level, as seen in Exhibit 34. Overall, middle-grades participants were 
significantly less likely to report positive feelings toward staff members than were 
participants in either elementary-grades or high school programs: 

■ Thirty-eight percent of middle-grades participants agreed a lot that 
staff always kept their promises, compared to 47 percent of 
elementary-grades participants and 48 percent of high school 
participants. 

■ Forty-seven percent of middle-grades youth felt that they could 
talk to staff about things that were bothering them, compared to 55 
percent of elementary-grades youth. (High school youth did not 
differ significantly in their responses from either group.) 

■ Forty-eight percent of middle-grades youth reported that staff 
cared what they thought, compared to 55 percent of high school 
youth. (Elementary-grades participants did not vary significantly 
in their responses.) 

■ Fifty-five percent of middle-grades youth agreed a lot that program 
staff helped them to try new things, compared to 64 percent of 
elementary-grades youth and 63 percent of high school 
participants. 

■ Fifty-five percent of middle-grades youth agreed a lot that staff 
really cared about them, significantly less than the 62 percent of 
elementary-grades youth and 61 percent of high school youth. 
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■ Sixty-three percent of middle-grades youth agreed a lot that staff 
thought they could learn new things, compared to higher percents 
among youth in elementary and high school programs (68 percent 
and 69 percent, respectively). 

■ Sixty-five percent of participants in elementary-grades programs 
agreed a lot that staff thought that they could do things well, 
compared to 60 percent of participants in middle-grades programs. 



Exhibit 34 

Participant Reports of Interactions with Staff, by Grade Levei 



In this program... 

Staff think I can learn new 
things’® 

Staff treat me w ith respect 
Staff think I can do things w ell* 

Staff help me to try new things’® 

Staff alw ays try to be fair*'' 

Staff really care about me*® 

I feel that I can talk to staff about 
things that are bothering me* 

Staff care w hat I think® 

Staff always keep their 
promises*® 
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Percent who "agreed a lot" 

■ High School participants (n=747) □ Middle-grades participants (n=1,031) □ Bementary-grades participants (n=1,795) 

* Elementary-grades programs differ significantly (p < .05) from middle-grades programs. 

° Middle-grades programs differ significantly (p < .05) from high school programs. 

^ Elementary-grades programs differ significantly (p < .05) from high school programs. 




In addition, elementary-grades students were significantly more likely to 
report that staff always tried to be fair than were either middle-grades or high 
school participants (64 percent, compared to 56 percent and 57 percent, 
respectively). 

Participants in center-based programs were more likely to report certain 
types of positive interactions with staff. For instance, 73 percent of youth in 
center-based programs reported that staff treated them with respect, compared to 
66 percent of youth in school-based programs. Sixty-six percent of participants in 
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center-based programs felt that staff really eared about them, eompared to 58 
pereent of partieipants in sehool-based programs. Partieipants in eenter-based 
programs were more likely to report that staff really eared what they thought, 
eompared to participants in school-based programs (55 pereent, eompared to 49 
pereent). Sixty-eight percent of youth in eenter-based programs reported that staff 
thought they eould do things well, eompared with 62 pereent of youth in sehool- 
based programs. Finally, 70 pereent of partieipants in eenter-based programs 
agreed a lot that staff thought they eould learn new things, eompared to 66 pereent 
of partieipants in school-based programs. 

These differenees by program loeation were espeeially notable for high 
sehool participants. In general, high sehool youth in sehool-based programs were 
signifieantly less likely to report that they had positive interactions with program 
staff For example, 65 percent of sehool-based partieipants believed that program 
staff treated them with respeet, eompared to 74 pereent of eenter-based 
partieipants. Fifty- five pereent of school-based youth felt that staff really eared 
about them, eompared to 68 pereent of eenter-based youth. Sixty pereent of 
sehool-based youth reported that staff thought they eould do things well, 
eompared to 68 pereent of eenter-based youth. Similarly, half of sehool-based 
partieipants (50 pereent) said that staff eared what they thought, eompared to 62 
pereent of eenter-based partieipants. Sehool-based partieipants were also less 
likely to believe that staff always kept their promises with only 44 pereent 
agreeing a lot. In eomparison, 52 pereent of eenter-based high sehool partieipants 
agreed. 



Healthy Living Skiiis 

Engaging in healthy lifestyles. Overall, more than half (53 pereent) of all 
OST partieipants reported that they engaged in physieal aetivities three hours a 
week or more. Twenty-nine pereent reported that they exereised five hours or 
more a week, and 24 pereent reported that they exercised three to five hours per 
week. An additional 25 pereent of youth reported that they exereised one to three 
hours a week. Middle-grades partieipants were somewhat more likely to report 
exercising on a regular basis, with 57 percent exercising three hours per week or 
more, eompared to 56 pereent of high school participants and 50 percent of 
elementary-grades partieipants. Responses did not vary by program loeation. 

Opportunities for youth leadership. In a set of questions asked of middle- 
grades and high school youth, participants in high school programs were more 
likely to report that they played leadership roles in the OST program than were 
partieipants in programs serving the middle grades, as seen in Exhibit 35. In 
partieular, 67 pereent of high sehool youth reported that they had been asked by 
staff for their ideas about the program or an aetivity, eompared to 56 percent of 
middle-grades youth. Fifty-eight pereent of high sehool partieipants reported that 
they had led an aetivity in the program, eompared to 5 1 pereent of middle-grades 
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participants. Similarly, high school participants were more likely to report that 
they had helped plan a program activity or event than were middle-grades 
participants (57 percent, compared to 37 percent). More than half (54 percent) of 
high sehool participants reported that they had helped out in the office, eompared 
to 26 percent of middle-grades participants. Forty-nine pereent of high school 
youth reported that they helped out on a youth council, advisory group, or 
leadership team for the program, compared to 30 pereent of middle-grades youth. 
Thirty-nine pereent of high sehool partieipants reported that they helped with 
meetings for parents or eommunity members, eompared to 20 pereent of middle- 
grades participants. 



Exhibit 35 

Participant Reports of Opportunities for Youth Leadership, 

by Grade Levei 




10. Youth Outcomes in Content Areas and 
Academic Skills 

OST programs promote many types of skills and knowledge, depending 
on the organizational foeus of their sponsoring organization and the skills of their 
staff Aeeording to the youth-development literature, it is important for youth to 
have opportunities to immerse themselves in positive, challenging skill-building 
activities, whatever the activity focus may be. These efforts teach important life 
lessons about the value of hard work in attaining an important goal, the value of 
eollaboration, and the personal satisfaction that comes from mastery of valuable 
skills. In its first year, the evaluation measured the development of program- 
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content skills through participant-survey items that address skill development in 
the OST program. The evaluation will examine and report on educational 
outcome data on samples of OST participants in Years 2 and 3 of the initiative to 
assess student outeomes in academic skills. These data will be obtained from the 
central student files of the DOE. 



Academic Benefits of the OST Program 

In interviews, youth described how the interactive learning environment 
and personal attention of the OST programs contributed to their aeademic 
improvements. For example, one participant described, “At first I was still a good 
student, but I failed my tests. Then I started getting up to it here [in the program]. 
My teaehers [in the program] make sure I understand. [You have to] be brave 
enough to tell your teaehers that you don’t understand. I get more attention here.” 
Another participant said, “We learn how to multiply fraetions, and do a number of 
word problems. It’s just fun because we go to the board and show our answers, 
and its fun because we get to eorrect [our answers] together, so someone can 
know how to do the problem the next time.” 

Evaluators asked youth to report on a series of items measuring the 
academic benefits of participating in an OST project. In general, partieipants in 
elementary-grades programs were significantly more likely to report high levels 
of aeademic benefits, eompared to either middle -grades or high school 
participants, as shown in Exhibit 36. The majority of elementary-grades 
participants (67 percent) agreed a lot that the program helped them finish their 
homework more often, signifieantly more than the 58 pereent of middle-grades 
partieipants and 33 percent of high sehool participants. Elementary-grades youth 
were also signifieantly more likely to report that the program helped them feel 
better about their sehoolwork, compared to middle-grades youth and high sehool 
youth (53 percent, compared to 35 percent and 31 pereent respectively). 

Similarly, 52 percent of elementary-grades youth reported that the program 
helped them solve math problems better and get better grades in sehool. Only 37 
pereent of middle-grades youth agreed with eaeh item. Twenty-five percent of 
high school youth reported that the program helped them solve math programs 
better, and 32 percent agreed that the program helped them get better grades in 
school. In addition, more than half of elementary-grades participants (5 1 percent) 
felt that the program helped them read and understand better, compared to only a 
third of middle-grades and high school participants (34 percent and 32 pereent). 
Forty-five pereent of elementary-grades youth reported that the program helped 
them write better, eompared to 33 pereent of middle-grades youth and 32 pereent 
of high school youth. 

One exception to this pattern is that high school participants were 
significantly more likely than elementary-grades participants to report that 
attending the program helped them learn to use computers to do sehoolwork better 
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(35 percent, compared to 29 percent). Middle-grades youth did not differ 
significantly from either the older or younger participants on this item. 



Exhibit 36 

Participant Reports of Academic Benefits, by Grade Levei 



This program has helped me... 




* Elementary-grades programs differ significantly (p < .05) from middle-grades programs. 
° Middle-grades programs differ significantly (p < .05) from high school programs. 

'' Elementary-grades programs differ significantly (p < .05) from high school programs. 



With two exceptions, findings of academic benefits did not vary 
significantly by program location. First, school-based participants were more 
likely to report that the program helped them solve math problems better, 
compared to center-based participants (44 percent, compared to 36 percent). 
Second, 60 percent of school-based participants reported that the program helped 
them finish their homework more often, compared to 48 percent of center-based 
participants. 



Academic Self-Esteem and Aspirations 

Overall, evaluators found notable differences in the academic self-esteem 
of OST participants by grade level, with academic self-esteem decreasing 
significantly between each grade cohort, as shown in Exhibit 37. This pattern was 
also seen in the TASC evaluation (Reisner et ah, 2004). Elementary-grades 
students reported significantly higher levels of academic self-esteem on the 
following measures, compared to both middle-grades and high school 
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participants; in addition, middle-grades partieipants had signifieantly more 
positive responses on these same items, compared to the high school participants: 

■ Eighty-one pereent of youth in elementary-grades programs 
reported that they tried hard in sehool, eompared with 68 pereent 
of youth in middle-grades programs and 5 1 pereent of youth in 
high sehool. 

■ Sixty-six pereent of partieipants in elementary-grades programs 
reported that they did well in sehool, compared to 54 pereent of 
participants in middle-grades programs and 39 percent of high 
sehool partieipants. 



Exhibit 37 

Participant Reports of Academic Seif-esteem, by Grade Levei 



In general I... 

Try hard in school**' 

Do well in school**' 

Bijoy math**' 

Pay attention in class**' 

Always finish my homework**' 

Always come to class prepared**' 

Enjoy w riting**' 

Enjoy reading books for pleasure*'^ 

Bijoy school’^' 
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■ High School participants (n=739) □ Middle-grades participants (n=1,018) □ Bementary-grades participants (n=1,779) 



* Elementary-grades programs differ significantly (p < .05) from middle-grades programs. 
° Middle-grades programs differ significantly (p < .05) from high school programs. 

^ Elementary-grades programs differ significantly (p < .05) from high school programs. 



■ In addition, 64 percent of elementary-grades youth agreed a lot that 
they enjoyed math, eompared with 45 pereent of middle-grades 
youth and 34 percent of high school youth. 

■ Sixty-four percent of partieipants in elementary-grades programs 
reported that they paid attention in elass, compared to 49 pereent of 
participants in middle-grades programs and 42 percent of high 
school participants. 
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■ Elementary-grades participants were more likely to report that they 
always finished their homework, compared to middle-grades 
participants and high school participants (63 percent, 47 percent, 
and 33 percent, respectively). 

■ Similarly, 63 percent of youth in elementary-grades programs said 
that they always come to class prepared, compared to 49 percent of 
youth in middle-grades programs and 42 percent of high school 
youth. 

■ Finally, 58 percent of participants in elementary-grades programs 
reported that they enjoy writing, compared with 42 percent of 
participants in middle-grades programs and 35 percent of high 
school participants. 

In addition, elementary-grades students were significantly more likely to 
report academic motivation in the following areas than their older peers. There 
were no significant differences on these items between middle-grades and high 
school students. 

■ Fifty-seven percent of elementary-grades students reported that 
they enjoy school, compared to 38 percent of middle-grades 
students and 33 percent of high school students. 

■ Elementary-grades youth were more likely to report that they 
enjoyed reading books for pleasure, compared to middle-grades 
youth, and high school youth (57 percent, compared to 32 percent 
and 33 percent). 

Participants in school-based programs reported higher levels of academic 
self-esteem, compared to participants in center-based programs. Evaluators 
speculate that these differences reflect the personal characteristics of youth who 
chose school-based and center-based programs, with students who enjoy school 
being especially likely to enroll in school-based programs. For example, 74 
percent of youth in school-based programs said that they tried hard in school, 
compared with 63 percent of youth in center-based programs. Participants in 
school-based programs were more likely to report that they paid attention in class, 
compared to participants in center-based programs (56 percent, compared to 51 
percent). Similarly, 56 percent of participants in school-based programs reported 
that they always came to class prepared, compared to 49 percent of participants in 
center-based programs. Forty-eight percent of youth in school-based programs 
enjoyed school, compared to 41 percent of youth in center-based programs. 
School-based participants were more likely to report that they enjoyed reading 
books for pleasure, compared to center-based participants (46 percent, compared 
to 42 percent). In addition, 54 percent of school-based participants reported that 
they enjoyed math, compared with 46 percent of center-based participants. More 
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than half (54 percent) of school-based participants said that they always finished 
their homework, compared to 47 percent of center-based participants. Finally, 
school-based participants were somewhat more likely to agree a lot that they do 
well in school, compared to center-based participants (58 percent, compared to 52 
percent). 

Differences in academic self-esteem were especially notable for high 
school participants in school-based programs, compared to those in center-based 
programs. In general, high school participants in school-based programs were 
more likely to report high levels of academic self-esteem than were their peers in 
center-based programs. For example, 57 percent of high school youth in school- 
based programs reported that they tried hard in school, in comparison with 45 
percent of youth in center-based programs. Thirty-six percent of school-based 
participants reported that they try to enjoy school, compared to 29 percent of 
center-based participants. Finally, high school youth in school-based programs 
were also more likely to report that they enjoyed reading books for pleasure, 
compared to high school youth in center-based programs (37 percent, compared to 
29 percent). 



Educational Aspirations 

The survey asked middle-grades and high school youth to report how sure 
they were that they would finish high school and how much further they would 
like to go in school. Seventy-six percent of all participants reported that they 
were very sure that they would finish high school. Forty-four percent of all 
responding youth reported that they would like to finish college. Participants in 
middle-grades programs were more likely to report that they would like to finish 
college, compared to high school participants (47 percent, compared to 41 
percent). Participants in center-based programs were more likely to report that 
they were very sure they would finish high school, compared to participants in 
school-based programs (81 percent, compared to 74 percent). Similarly, 44 
percent of center-based participants reported that they would like to get more 
education after college, compared to 36 percent of school-based participants. 



Development of Other Competencies 

In addition to questions about academic competencies, the survey also 
asked youth about development of other cognitively related skills. Across all 
grade levels, 61 percent of participants agreed a lot that they could play a musical 
instrument, sing, dance, draw, paint, or could do some other type of art really 
well. Sixty percent of participants said that they were good at using a computer. 
More than half (54 percent) reported that they had the skills to be a leader. 
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These responses varied by grade level, with younger students signifieantly 
more likely to report strong skills than older partieipants. In partieular, 
elementary-grades partieipants were signifieantly more likely to report that they 
were good at using a eomputer than either middle-grades or high school 
participants (64 percent, compared to 59 percent and 48 percent). Fifty-eight 
percent of elementary-grades youth reported that they had the skills to be a leader, 
followed by 54 percent of middle-grades youth and 45 percent of high school 
youth. When asked whether or not they could play a musical instrument, sing, 
dance, draw, paint, or could do some other type of art really well, 68 percent of 
elementary-grades youth agreed a lot, compared to 59 percent of middle-grades 
youth and 46 percent of high school youth. Evaluators speculate that these 
differences reflect the greater knowledge and sophistication of older youth, who 
are more likely to have learned how much there is to know. 

School-based participants were significantly more likely to report that they 
were good at using a computer, compared to center-based participants (61 
percent, compared to 56 percent). In addition, 62 percent of school-based youth 
reported that they could play a musical instrument, sing, dance, draw, paint, or 
could do some other type of art really well, compared to 56 percent of center- 
based youth. 

Among middle-grades and high school participants, 48 percent of 
respondents said that they knew about different careers. Thirty-eight percent 
reported that they were comfortable speaking in front of a group. Thirty-seven 
percent reported that they had a strong attachment to their neighborhood. Thirty- 
four percent felt that they could make a difference in their neighborhood. These 
results differed by grade level. When asked whether they knew about different 
careers, 5 1 percent of middle-grades youth agreed a lot, compared to 44 percent 
high school youth, again perhaps reflecting the greater awareness of older youth 
of how much they had yet to learn. Middle-grades youth were also more likely to 
report that they had a strong attachment to their neighborhood (43 percent, 
compared to 29 percent). Thirty-seven percent of middle-grades youth felt that 
they could make a difference in their neighborhood, compared to 30 percent of 
high school youth. 



11. Associations between Program Features and 
Participant Experiences 

Over the three years of the evaluation, evaluators will seek to determine 
whether specific program characteristics influence participants’ engagement in the 
OST program and participants’ social development and academic outcomes. The 
design of the evaluation does not permit evaluators to identify causal influences, 
but it does permit the calculation of statistically relevant associations between 
program features and participant reports in important areas of youth experience. 
Exploring these early associations between program features and participant 
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experiences can help the OST initiative identify elements of program 
implementation that should be emphasized or monitored with special care. These 
associations can also help sponsoring organizations and program directors to 
make decisions about program characteristics that may shape after-school 
effectiveness. 

To measure these associations, evaluators developed survey scales that 
combine participant responses to a series of items related to the same outcome 
into a single measure. These scales were computed to range from one to four, 
with four indicating that on average participants strongly agreed with the series of 
statements related to that scale. The evaluation also created variables measuring 
program-level content focus, based on program directors’ survey responses 
regarding the frequency and availability of activities related to various content 
areas. See Appendix E for technical details regarding correlation coefficients and 
significance levels for each of the associations examined. 



Associations with Academic Outcomes 

In Year 1 of the evaluation, findings of associations between student 
outcomes and program characteristics were concentrated in student reports of 
academic self-esteem and academic benefits of the program. Across all Option I 
programs: 



■ On average, participants who attended programs with a strong 
focus on academic activities reported more academic benefits of 
the OST program (correlation of .37). Also, academic self-esteem 
was positively correlated with the amount of academic activities 
offered by the program (correlation of .25). 

■ The degree to which programs offered arts activities was positively 
associated with participant reports of academic benefits 
(correlation of .23) and academic self-esteem (correlation of .23). 

Evaluators also found a positive association between average participant 
reports of academic self-esteem and OST program participation rates, as 
measured by the percent of days participants attended the activities to which they 
were assigned (correlation of .38). This association was particularly strong for 
school-based programs (correlation of .48). This finding may imply a self- 
selection factor in OST program participation, with students who have a stronger 
feeling of self-efficacy in school attending the program more regularly, or it could 
suggest that OST program participation contributes to improved academic self- 
esteem. This pattern and others will be explored in future years of the OST 
evaluation. 
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Associations with Sociai Deveiopment Outcomes 

In contrast to the academic outcomes finding, there was a negative 
association between participants’ reports of the quality of their interactions with 
OST program staff members and the degree to which the program offered 
academic activities (correlation of -.23). 

In Year 1 of the OST initiative, there is also some evidence that 
participants in programs that focus on providing a high intensity of physical 
fitness activities (including structured and unstructured sports practice or games) 
reported lower levels of satisfaction with the ways in which participation exposed 
them to new and engaging experiences in the program and less positive 
interactions with OST staff members (correlations of -.25 and -.32, respectively). 
Later years of the evaluation will explore whether physical fitness activities in 
these programs may be crowding out other types of experiences for youth. 



12. Systems Outcomes 

The OST initiative aims to strengthen the city’s capacity to serve youth 
and meet the needs of working families. To address these goals, the evaluation 
collected data to assess the extent to which the initiative is (1) increasing the 
capacity of provider organizations to deliver high-quality OST services, 

(2) increasing the capacity of private nonprofit providers and public agencies to 
function as a coherent system focused on improving OST opportunities for 
children and youth, and (3) meeting the needs of working parents. 



Impact of OST Initiative on Provider Organizations 

In Year 1, OST contracts were awarded to nearly 200 provider 
organizations in New York City that varied considerably in their scope and 
experience in providing out-of-school time services for youth. Executive 
directors most frequently described their organizations as youth-serving 
organizations (33 percent) or as social service organizations (24 percent). 

Overall, provider organizations emphasized social development objectives 
for their OST programming. More than three-quarters of executive directors 
identified the following as major objectives for their programs: 

■ Provide a safe environment for participants (91 percent) 

■ Help participants develop socially (90 percent) 

■ Provide participants with positive adult guidance and/or mentors 
(83 percent) 
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In addition, 74 percent of executive directors identified helping participants 
improve their academic performance as a major objective. 

Out-of-school time budgets. For the most part, the OST provider 
organizations were well established with sizable budgets: 52 percent of executive 
directors reported that their annual organizational budget was more than $3 
million, an additional 10 percent had an annual budget of more than $2 million, 
and 12 percent had an annual budget of less than $500,000. Provider 
organizations also reported large budgets for their out-of-school time 
programming. During 2005-06, the total out-of-school time budget for 50 percent 
of organizations was more than $500,000, including the DYCD OST grant(s). It 
was more than $1 million in 34 percent of organizations. Only 15 percent of 
organizations had a total out-of-school time budget of less than $100,000 during 
the 2005-06 year. 

DYCD OST funding played a major role in the out-of-school time budgets 
of provider organizations. On average, organizations drew 59 percent of their 
OST budgets from DYCD funding. Twenty percent of organizations relied 
exclusively on DY CD funds for their out-of-school time programming budget. 
Forty-eight percent of executive directors reporting drawing at least half of their 
out-of-school time budgets from DYCD funds. Provider organizations 
supplemented OST initiative funds primarily with general organizational funds 
(on average, 16 percent of out-of-school time budgets were drawn from general 
organizational funds). Organizations relied less on other city or state sources, 
which on average comprised 6 percent each of organizations’ out-of-school time 
budgets. 

The DYCD OST initiative also played a major role in the staff hired by 
provider organizations. Executive directors reported that their organizations on 
average employed 292 staff members. On average, staff members working in 
OST programs represented 30 percent of the organizations’ total employees. The 
median number of employees per organization was 123. 

Opportunities available to provider organizations. Evaluators also asked 
executive directors about the extent to which the OST initiative had increased the 
opportunities available to their organization. Overall, as shown in Exhibit 38, 
executive directors reported that participation in the DYCD OST initiative had the 
most effect on their opportunities for training and technical assistance to staff: 74 
percent responded that DYCD participation increased their organizations’ 
opportunities to participate in training either “to a great extent” or “somewhat.” 
Eor the majority of providers, DYCD participation also increased their 
opportunities to partner with city agencies (62 percent), to partner with a public 
school (58 percent), or to partner with cultural organizations (55 percent). More 
than one-third of executive directors reported that DYCD participation had 
increased their opportunities to access city funding sources other than DYCD (38 
percent). 
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Exhibit 38 

increased Opportunities Avaiiabie to Provider Organizations 
Due to OST Initiative (n=159) 

The DYCDOST initiative has increased opportunities... 

For training and technicai assistance for staff 
To partner w ith city agencies 
To partner w ith a public schooi 
To partner w ith culturai organizations 
To access city funding sources other than DYCD 
To access corporate/private funding sources 
To access state funding sources 
To access federal funding sources 

0 20 40 60 80 100 

Percent who answered "to a great extent" or "somewhat" 




Challenges of the OST initiative. According to executive direetors 
surveyed, the greatest ehallenge of participating in the OST initiative for provider 
organizations was the administrative burden; 61 percent reported this to be a 
challenge, as shown in Exhibit 39. In addition, more than a third of executive 
directors identified the following ehallenges in the first year of the OST initiative: 

■ Using the OST Online data tracking system (54 percent) 

■ Meeting enrollment and attendance requirements (40 percent) 

■ Balaneing demands of OST with other organizational goals and 
priorities (38 pereent) 

■ Meeting timelines for program start-up (38 percent) 

■ Accessing funds (34 percent) 
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Exhibit 39 

Chaiienges of the OST Initiative to Provider Organizations (n=159) 



Administrative burden 
Using the OST Online tracking system 

Meeting enrollment and attendance requirements 

Balancing demands of OST w Ith other organizational goals 
and priorities 

Meeting timelines for program start-up 
Accessing funds 
Meeting city or state licensing requirements 
Integrating DYCD OST funds w ith other funding streams 
Securing appropriate space/faoilities 
Hiring qualified staff 

0 20 40 60 80 100 

Percent who answered "to a great extent" or "somewhat" 




Impact of OST Initiative on Youth Services in New York City 

Shift from ACS funding. The implementation of the OST initiative 
marked a change in child-care programs for school-age youth in New York City. 
As of September 2006, the Administration for Children’s Services (ACS) no 
longer contracted for child-care programs for school-age youth. Former ACS 
providers were encouraged to apply for funding through the OST initiative and to 
transition to the DYCD OST system. Option I OST programs were required to 
give enrollment priority to youth who were former participants in an ACS pre- 
school program or a school-age child-care program for youth under age 13. 

Twenty percent of executive directors surveyed reported that at least one 
of the programs their organization operated in 2004-05 was supported by ACS 
funds. Eighty-four percent of this subset of providers said that in the first year of 
the initiative, their DYCD OST program gave priority enrollment to former ACS 
participants. 

Effect of OST on scope of programming in NYC. The majority of 
provider organizations had extensive experience operating out-of-school time 
programs prior to the DYCD OST initiative. Eighty-seven percent of executive 
directors surveyed said that their organization had previously operated an after- 
school or out-of-school time program. Seventy percent of provider organizations 
had operated programs since at least the 1997-98 school year. 
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In addition, 73 percent of organizations continued to host at least one 
additional out-of-school time program that did not receive funding through the 
OST initiative in the 2005-06 school year; 53 percent operated a non-OST school- 
based program, and 42 percent operated a non-OST center-based program. 

Even so, the DYCD OST initiative permitted provider organizations to 
begin OST services in many new locations. Although 20 percent of Option I 
program directors reported that their program had been in operation for more than 
10 years before the OST initiative, nearly half (44 percent) reported that the 
initiative brought out-of-school time programming to their location for the first 
time. School-based programs were less likely to have offered out-of-school time 
programming prior to the DYCD OST grant than were center-based programs (51 
percent, compared to 28 percent). Center-based programs were more likely than 
school-based programs to have offered out-of-school time programming for more 
than 10 years (43 percent, compared to 10 percent). 

Seventy-seven percent of Option II programs directors reported that their 
programs existed prior to the DYCD OST grant. More than half (55 percent) had 
been in operation for six or more years. In contrast, nine of ten Option III 
program directors reported that out-of-school time programming was not offered 
prior to the DYCD OST grant. 

Effect of DYCD OST initiative on quality of out-of-school time services. 
Executive directors of provider organizations were asked about the ways that their 
OST-funded programs differed from other out-of-school time programs that the 
organization sponsored. As illustrated in Exhibit 40, 76 percent of executive 
directors reported that DYCD OST program staff tracked program attendance 
more than in their organization’s other out-of-school time programs. In addition, 
almost half (46 percent) of the executive directors reported that DYCD OST 
programs offered programming on weekends and holidays more than their other 
out-of-school time programs. 
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Exhibit 40 

Differences in OST Programming across DYCD 
and Other Sources of Support (n=139) 



Compared to the organization's other out-of-schooi time programs, OST programs... 



Track student program attendance 
Offer programming on weekends and holidays 
Offer regular, extensive programming hours 
Focus on academic programming 
Enforce minimum attendance policies for participants 
Focus on arts programming 
Comply w ith city and state child care regulations 
Offer summer programs 
Establish linkages with surrounding schools 
Provide referral services to participants and their families 
Have a low staff -to-youth ratio 
Focus on sports and recreation programming 
Adhere to strict standards for hiring/screening qualified staff 

aher 

Provide access for children w ith special needs 




0 20 40 60 80 100 



Percent who answered "somewhat more" or "much more" 



About one-third (34 percent) of executive directors reported that their 
organization’s DYCD OST programs offered academic programming more than 
their other out-of-school time programs. Thirty-four percent also reported that 
OST programs enforced minimum attendance policies for participants more than 
their other programs. 



Meeting the Needs of Working Parents 

A key goal of the OST initiative is to support the needs of working 
families. Indeed, the majority of parents who responded to the survey indicated 
that they were employed. More than three-quarters of responding parents (81 
percent) reported that they worked at least 20 hours per week, and more than half 
(57 percent) worked more than 35 hours per week. In addition, about a third of 
parents (34 percent) responded that they were enrolled in school. 

Reasons for enrolling their child. As illustrated in Exhibit 41, parents 
reported a host of reasons for enrolling their child in the OST program. The most 
frequently reported reasons included; 
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■ I believed the program would help my child do better in school (86 
percent) 

■ I wanted my child to participate in new types of activities (83 
percent) 

■ I wanted my child to be safe after school (82 percent) 

■ I wanted my child to get help with homework (79 percent) 



Exhibit 41 

Reasons for Enroiiing Chiid, According to Parents (n=276) 



I believed the program would help my child do better In school 
I wanted my child to participate in new types of activities 
I w anted my child to be safe after school 
I w anted my child to get help w ith homew ork 
The program is free 
I wanted my child to develop better personal/social skills 
My child w anted to enroll 
I wanted my child to gain more confidence in hinVherself 
It was a way for my child to be with friends 
I needed child-care 

0 20 40 60 80 100 

Percent who responded "a big reason" 




In total, nearly three-quarters of parents reported that the OST academic 
activities were most important to them; 43 percent of parents indicated that 
homework help was the most important activity offered, and 28 percent selected 
academic enrichment as the most important activity. 

Outreach to parents. Programs reached out to parents to varying degrees. 
Across all Option I programs, about one third (35 percent) reported having a 
parent liaison or parent outreach coordinator, including 14 percent with a 
volunteer coordinator, 14 percent with a paid part-time liaison, and 7 percent with 
a paid full-time parent liaison or outreach coordinator. Program directors reported 
most frequently communicating with parents through phone conversations and 
meetings with one or more parents (55 and 46 percents, respectively, reported 
engaging in these types of outreach at least weekly). 

OST programs also offered activities to engage parents, inform them about 
services available, or increase parents’ skills, as shown in Exhibit 42. Program 
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directors most frequently reported offering parents opportunities to attend events 
at the program and to attend cultural or recreational events in the community (90 
percent and 75 percent, respectively). 



Exhibit 42 

Activities Offered for Parents, 
According to Option I Program Directors (n=390) 



B/ents at the program 

Opportunities to attend cuiturai or recreational events in the 
community 

Opportunities to hear from and taik w ith representatives 
from locai agencies or other organizations 

Parenting classes 
Gasses to help parents develop their ow n skills 
Biglish as a Second Language classes 

0 20 40 60 80 too 

Percent who reported ever offering the activity 




Parent satisfaction. Overall, parents were satisfied with the quality of the 
OST program their child attended, according to the small number of parents 
surveyed in the evaluation. Sixty-two percent of parents rated the overall quality 
of their programs as excellent. 

As illustrated in Exhibit 43, parents were very satisfied with their OST 
program’s ability to provide a safe space for students to participate in activities 
and interact with other youth. Sixty-seven percent of parents strongly agreed that 
their child was able to join activities that they would not have attended otherwise, 
67 percent felt their child was safer in the out-of-school hours as a result of the 
program, and 62 percent reported that their child made new friends. 
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Exhibit 43 

Parent Reports of Program Benefits (n=281) 



As a result of participating in the program. . . 

My child is able to join activities that she or he w ould not 
be able to otherw ise attend 

My child is safer in the out-of-school hours 
My child is making new friends 
My child is more confident 
My child is getting the academic help he/she needs 
My child talks to me more about w hat's going on in school 
My child is doing better in school overall 
My child is behaving better at home 

0 20 40 60 80 100 

Percent who "strongly agreed" 




In addition, about half of responding parents strongly agreed that their 
ehild benefited academically from participation in the OST program. Fifty-five 
percent strongly agreed that their child was getting the academic help he/she 
needed, 54 percent felt that their child talked to them more about what was going 
in school, and 54 percent strongly agreed that their child was doing better in 
school overall as a result of participating in the after-school program. 

As illustrated in Exhibit 44, parents also expressed satisfaction in how 
well the OST program met their own needs. Seventy-one percent of parents 
strongly agreed that the program hours fit their needs, 63 percent reported that 
they missed less work than they used to, and 61 percent reported that the program 
made it easier for them to keep their job. 
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Exhibit 44 

Parent Reports of How Program Met their Needs (n=265) 



The program hours fit my needs 
I miss iess work than i used to 
The program has make it easier to keep my job 
I am able to work more hours 



I have been able to go back to w ork 
The program has made it easier for me to go to school 

0 20 40 60 80 too 

Percent who "strongly agreed" 




13. Conclusions 

These evaluation findings indicate that overall the DYCD OST initiative 
was successful in launching a broad scope of out-of-school time programming 
opportunities in New York City in its first year. These emerging findings also 
suggest avenues for improving the effectiveness of OST programming as the 
initiative matures over the next few years. 

Programs successfully enrolled students but struggled to maintain 
high youth participation rates. Participants must be exposed to high-quality 
programming on a regular basis to benefit in measurable ways from participation. 
Although OST programs enrolled an impressive 50,000-plus participants in more 
than 500 programs throughout New York City in the initiative’s start-up year, 
student-level attendance rates were modest: in elementary- and middle-grades 
programs, only a quarter of participants received the minimum threshold of hours 
of programming sought by DYCD, as did about a third of high school 
participants, according to data available to the evaluation. This finding suggests a 
need to establish program policies and create activity offerings that encourage 
youth to attend OST programs on a frequent basis in order to maximize the 
benefits of participation. In addition, the evaluators note that some of the 
apparent attendance issues may actually reflect program-level deficiencies in 
keeping activity-participation data current in OST Online. 

Programs consistently provided safe and structured environments for 
participants in the out-of-school hours. Particularly in OST programs serving 
elementary- and middle-grades youth, in the first year of the initiative program 
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directors focused on providing a safe environment for youth and on providing 
academic support, primarily through recreation and homework help. As a result, 
participants were more likely to report feeling connected and comfortable in their 
OST programs than they were to report that the program exposed them to new 
opportunities. As program operations and staffing patterns become well 
established, the challenge will be to provide creative learning opportunities that 
engage youth and retain them in the OST program. Programs targeting high 
school students were most effective at providing interactive activities that 
appealed to youth, perhaps reflected in the finding that high school youth on 
average attended their OST program for more hours than expected by the 
initiative. 

Hiring program staff members who were well qualified to provide 
high-quality OST programming was a challenge for OST programs in Year 

1 . Program directors all reported that they were challenged to find qualified staff 
to hire. Limited resources available to pay staff also emerged as a major 
challenge. Indeed, staff salaries typically averaged only $6 to $1 1 per hour. This 
finding suggests that OST programs will need to develop effective staffing 
patterns and structures to improve the skills of their young staff members through 
professional development opportunities and by mentoring from more experienced 
program staff members. 

Varying program structures used by center- and school-based 
programs may be contributing to varying outcomes. Findings from Year 1 of 
the evaluation indicate that participants in center-based programs felt a 
particularly strong sense of connection with their OST program and reported more 
positive interactions with staff members, likely precursors to other social 
development outcomes. Participants in school-based programs tended to report 
stronger academic benefits from the program. Identifying key features of each 
type of program will help all programs improve both the social and academic 
supports provided to OST participants. 

Over the next two years, the evaluation of the OST initiative will continue 
to collect data from OST programs to explore these emerging patterns. Future 
reports will examine changes in program implementation and participants’ social 
and academic outcomes as the programs become more fully established in their 
schools and communities. 
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Appendix A 

Priority Middle School OST Programs 



In Year 1 of the OST initiative, DYCD funded 15 programs that were 
intended to target students in high-need middle schools and to operate in 
collaboration with state-approved Supplemental Educational Services providers. 
The evaluation collected program director and participant survey data from these 
Priority Middle School programs. Program experiences are highlighted below. 

Program implementation. According to the program director survey 
responses, Priority Middle School programs most regularly offered homework 
help (14 of 15 programs), organized team sports instruction/activities (13 
programs), group instruction in specific academic subjects (12 programs), and 
recreational reading (11 programs). 

Reflecting their location in high-need middle schools, 1 1 Priority Middle 
School OST program directors reported that they specifically sought to serve 
youth identified by their school as needing special assistance in reading and/or 
math, and 1 1 served youth who were recommended by school-day teachers or 
counselors. Eleven of the program directors also reported that they allowed open 
enrollment for all interested youth. The program directors communicated 
relatively frequently with school-day staff. At least monthly: 14 of 15 reported 
that they discuss student discipline policies with school staff; 13 discussed the 
needs or progress of individual students; and 12 reported that they discussed 
homework assignments and planning of OST content with school staff. 

Participant experiences. Participants generally reported strong 
attachment to their program. More than half “agreed a lot” with the following 
statements about the OST program: 

■ lam safe (63 percent) 

■ This is a good place to hang out (62 percent) 

■ Staff treat me with respect (62 percent) 

■ Staff think I can learn new things (59 percent) 

■ I belong (56 percent) 

■ The activities really get me interested (52 percent) 

Priority Middle School program participants also reported relatively high 
levels of academic motivation. More than half “agreed a lot” with the following 
statements: 



■ I try hard in school (76 percent) 

■ I am sure I will graduate from high school (74 percent) 

■ I do well in school (56 percent) 

■ I always try to come to class prepared (56 percent) 
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However, they reported only moderate levels of aeademic benefits from 
partieipating in the OST program. Most notably, 47 percent “agreed a lot” that 
the program helped them finish their homework more often. 



Comparison of Priority Middie Schooi Programs and Other Option I 
Middie Schooi Programs 

Participant experiences varied somewhat between the Priority Middle 
School Programs and other OST Option I middle-grades programs. As displayed 
in the exhibits below, participants reported statistically significant differences in 
the following areas: 

■ Priority Middle School participants were less likely to report that 
the OST program gave them a chance to do a lot of new things or 
that there was a lot to choose to do than were other middle-grades 
participants (53 percent compared to 46 percent, and 52 percent 
compared to 44 percent, respectively) 

■ Priority Middle School participants were significantly less likely 
than other middle-grades participants to report feeling safe in the 
program (7 1 percent, compared with 63 percent) or to feel that 
their ideas count in the program (44 percent, compared to 38 
percent) 

■ Compared with other middle-grades participants. Priority Middle 
School participants were less likely to report positive interactions 
with staff. In particular, fewer Priority Middle School participants 
agreed a lot that staff think they can do things well (60 percent, 
compared to 55 percent), that staff always try to be fair (56 
percent, compared to 5 1 percent), and that staff really care about 
them (55 percent, compared to 48 percent) 
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Exhibit A1 

Middie-Grades Participant Reports of Exposure to New Experiences 



In this program... 



The activities reallygetme interested 



I get a chance to do a lot of new things* 



There is a iotfor me to choose to do* 



i get to do things that i don't usuaily get to do anywhere eise 



i get to work on projects that reailymake me think 



0 20 40 60 80 too 

Percent who "agreed a lot" 

□ Ali other Option i middie schooi participants (n=1 ,029) 

■ Priority Miodie School participants (n=558) 

* p< .05 




Exhibit A2 

Middie-Grades Participant Reports of Sense of Beionging 



In this program I feel like... 



1 am safe* 




|71 




This is a good place to hang 




160 


out 




1 belong 




o 

CD 


1 am successful 




00 

LO 


1 matter 




1 56 




My ideas count* 


144 



0 20 40 60 80 too 

Percent who "agreed a lot" 



□ All other Option I middle school participants (n=1 ,026) 
■ Priority Middle School participants (n=548) 



* p< .05 
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Exhibit A3 

Middie-Grades Participant Reports of Interactions with Staff 



In this program... 



Staff treat me with respect 




- 




Staff think 1 can learn new things 




- 




Staff help me to try new things 




- 




Staff think 1 can do things well* * 




- 




Staff always try to be fair* 




- 




Staff really care about me* 




- 




feel that 1 can talk to staff about things that are bothering me 




- 




Staff care what 1 think 




- 




Staff always keep their promises 







0 20 40 60 80 100 

Percent who "agreed a lot" 



□ All other Option I middle school participants (n=1 ,031) 

■ Priority Middle School participants (n=549) 



* p< .05 



However, participants in Priority Middle Schools were significantly more 
likely to report protecting another student from a bully than were other middle- 
grades participants (71 percent, compared with 59 percent). Priority Middle 
School participants were also significantly more likely to report having helped 
plan a program activity or event than were other middle-grades participants (42 
percent, compared with 37 percent) 
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Exhibit A4 

Prosociai Behaviors among Middie-Grades Participants 



In the past 30 days I... 



Helped other students solve a problem 
Cooperated with others in compieting a task 
Gave someone a compiiment 

Toid other students how I feit when they upset me 

Toid other students how I feit when theydid something I 
liked 

Protected someone from a bully* 
Helped someone stay out of a fight* 




Percent who reported doing the activity at least once 



* p< .05 



□ All other Option I middle school participants (n=1 ,026) 
■ Priority Middle School participants (n=544) 



Exhibit A5 

Middie-Grades Participant Reports of Opportunities for 
Youth Leadership 



At this program, I have... 




□ All other Option I middle school participants (n=1,027) 
■ Priority Middle School participants (n=545) 



* p < .05 
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Differences in participant reports of academic benefits were mixed. 
Priority Middle School participants were less likely than other middle-grades 
participants to agree a lot that the program helped them finish their homework 
more often (47 percent compared to 58 percent), but more likely to report that the 
program helped them to use computers to do schoolwork better (37 percent, 
compared to 30 percent). 



Exhibit A6 

Middie-Grades Participant Reports of Academic Benefits 



This program has helped me... 



Finish my homework more often* 
Use computers to do schoolwork better* 
Read and understand better 
Solve math problems better 
Get better grades in school 
Feel better about myschoolwork 
Write better 




Percent who "agreed a lot" 



□ All other Option I middle school participants (n=1 ,032) 
■ Priority Middle School participants (n=549) 



* p< .05 



There were no significant differences in terms of reports of academic self- 
esteem or educational aspirations between Priority Middle School participants and 
other middle-grades participants. 
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Appendix B 

Scales Used in Program Observation Analyses 



In Year 1 of the evaluation of the OST initiative, evaluators eonducted 
between 10 and 12 activity observations in each of 15 in-depth study sites. 
Evaluators used PSA’s OST Observation Instrument to conduct these structured 
15 minute observations. In total, observation data represent 238 independent 
observations and 40 activity co-observations with an inter-rater reliability of 0.83. 

Relationships 

The Relationships scale combines ratings from the following indicators; 

■ Y outh are friendly and relaxed with one another 

■ Y outh respect one another 

■ Y outh show positive affect to staff 

■ Y outh are on-task 

■ Y outh listen actively and attentively to peers and staff 

■ Staff use positive behavior management techniques 

■ Staff show positive affect toward youth 

■ Staff attentively listen to and/or observe youth 

■ All or most youth take leadership responsibility/roles 

Descriptive Statistics: 



Alpha 


Mean 


Standard 

Deviation 


Minimum 


25* 

Pereentile 


75* 

Pereentile 


Maximum 


0.88 


5.31 


0.96 


1.78 


4.78 


6.00 


7.00 




Scale score 
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Instructional Strategies 



The Instructional Strategies scale combines ratings from the following indicators: 

■ Youth contribute opinions, ideas and/or concerns to discussions 

■ Staff encourage youth to share their ideas, opinions and concerns 

■ Staff communicate goals, purposes, expectations 

■ Staff verbally recognize youth’s efforts and accomplishments 

■ Staff assist youth without taking control 

■ Staff ask youth to expand upon their answers and ideas 

■ Staff challenge youth to move beyond their current level of 
competency 

■ Staff employ two or more teaching strategies 
Descriptive Statistics: 



Alpha 


Mean 


Standard 

Deviation 


Minimum 


25* 

Pereentile 


75* 

Pereentile 


Maximum 


0.87 


3.06 


1.46 


1.00 


2.00 


3.75 


6.63 




Scale score 
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Activity Content and Structure 



The Activity Content and Structure scale combines ratings from the following 
indicators; 



■ The activity is well organized 

■ The activity involves the practice/a progression of skills 

■ The activity challenges students intellectually, creatively, and/or 
physically 

■ The activity requires analytic thinking 
Descriptive Statistics: 



Alpha 


Mean 


Standard 

Deviation 


Minimum 


25* 

Pereentile 


75* 

Pereentile 


Maximum 


0.84 


4.26 


1.67 


1.00 


2.88 


5.75 


7.00 




1.49 1.99 2.49 2.99 3.49 3.99 4.49 4.99 5.49 5.99 6.49 7 

Scale score 
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Appendix C 

Details of Data Used in Analyses of Program Features 



Exhibit C1 

Types of Youth Targeted by Option I Programs for OST Services, 
According to Program Directors, by Grade Levei 



Percent of Program Directors 


Bonferroni Test 




Elementary-grades 
Program Directors 
(n=155) 


Middle-grades 
Program Directors 
(n=121) 


High School 
Program Directors 
(n=117) 


Comparison 


Sig. 


We have open enrollment for all 
interested youth 


92 


89 


92 




1.00 


Middle-High 


1.00 


Elementary-High 


1.00 


Youth who are recommended by 
school-day teachers or counselors 


62 


62 


51 


Elementary-Middle 


1.00 


Middle-High 


0.28 


Elementary-High 


0.23 


Youth with siblings already 
attending the program 


59 


49 


25 


Elementary-Middle 


0.26 


Middle-High 


0.00* 


Elementary-High 


0.00* 


Youth identified by their school as 
needing special assistance in 
reading and/or math 


48 


50 


30 


Elementary-Middle 


1.00 


Middle-High 


0.06* 


Elementary-High 


0.09* 


Youth who receive free- or reduced- 
price lunch 


45 


40 


21 




1.00 


Middle-High 


0.05* 


Elementary-High 


0.00* 


Youth who scored “below proficient” 
on city or state assessments 


43 


41 


27 


Elementary-Middle 


1.00 


Middle-High 


0.03* 


Elementary-High 


0.11 


Youth who are English-language 
learners 


43 


31 


19 


Elementary-Middle 


0.93 


Middle-High 


0.14 


Elementary-High 


0.00* 


Youth who are referred through our 
organization 


40 


40 


34 


Elementary-Middle 


1.00 


Middle-High 


1.00 


Elementary-High 


1.00 


Youth who participate in other 
programs sponsored by our 
organizations 


38 


37 


37 


Elementary-Middle 


1.00 


Middle-High 


1.00 


Elementary-High 


1.00 



* p < .05 



Exhibit reads: Ninety-two percent of eiementary-grades program directors reported that their programs have open 
enroiiment for aii interested youth, compared to 89 percent of middie-grades program directors and 92 percent of high 
schooi program directors. There were no significant differences by grade ievei. 
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Exhibit C2 

Types of Youth Targeted by Option I Programs for OST Services, 
According to Program Directors, by Program Location 



Percent of Program Directors 




School-based 
Programs (n=272) 


Center-based 
Programs (n=121) 


Chi-Square 


P 


We have open enrollment for all 
interested youth 


90 


93 


1.13 


0.29 


Youth who are recommended by 
school-day teachers or counselors 


63 


49 


7.24 


0.01 


Youth identified by their school as 
needing special assistance in 
reading and/or math* 


46 


36 


3.98 


0.05 


Youth with siblings already 
attending the program* 


45 


47 


0.17 


0.68 


Youth who scored “below proficient” 
on city or state assessments 


39 


33 


1.41 


0.24 


Youth who receive free- or reduced- 
price lunch 


37 


34 


0.30 


0.58 


Youth who are referred through our 
organization 


36 


44 


2.35 


0.07 


Youth who participate in other 
programs sponsored by our 
organizations* 


34 


45 


3.90 


0.05 


Youth who are English-language 
learners 


34 


21 


1.68 


0.20 



* p<0.05 



Exhibit reads: Ninety percent of directors of schooi-based programs reported that their program had open 
enroiiment for aii interested youth, compared to 93 percent of directors of center-based programs. This 
difference was not statisticaiiy significant. 
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Exhibit C3 

Option I Program Director Prior Experience, by Program Location 



Percent of Program Directors 




School-based 
Programs (n=250) 


Center-based 
Programs (n=116) 


Chi-Square 


P 


Camp counselor/leader 


56 


58 


0.20 


0.66 


Program director in an OST 
program* 


55 


69 


6.23 


0.01 


Recreation, youth, or child-care 
worker 


53 


60 


1.55 


0.21 


Staff member in an OST program* 


51 


62 


4.15 


0.04 


Instructional specialist 


42 


42 


0.00 


0.99 


Administrator at a child/youth center 
or at a park or recreation center 


41 


52 


3.82 


0.05 


Classroom teacher 


40 


45 


0.59 


0.44 


Administrator in a social services 
organization 


39 


48 


2.15 


0.14 


Pupil support staff 


26 


29 


0.21 


0.65 


Social services or health services 
provider 


25 


30 


0.93 


0.34 


Classroom aide/teaching assistant 


24 


27 


0.42 


0.52 


School administrator 


14 


16 


0.44 


0.51 



* p<0.05 



Exhibit reads: Fifty-six percent of directors of schooi-based programs reported having prior experience as 
a camp counseior/ieader, compared to 58 percent of directors of center-based programs. This difference 
was not statisticaiiy significant. 



Exhibit C4 

Communication with Schooi Staff, as Reported by 
Directors of Option I Programs, by Program Location 



Percent of Program Directors 
Reporting Communication at Least Once a Month 




School-based 
Programs (n=270) 


Center-based 
Programs (n=117) 


Chi-Square 


P 


The needs or progress of individual 
students* 


70 


52 


11.74 


0.00 


Issues related to 
classrooms/sharing space* 


63 


33 


28.33 


0.00 


Homework assignments* 


62 


41 


15.27 


0.00 


Planning OST program content* 


61 


35 


22.28 


0.00 


Student discipline policies* 


58 


34 


19.86 


0.00 


Curriculum concepts currently being 
taught in school* 


49 


35 


6.16 


0.01 


OST program enrollment policies* 


39 


25 


6.71 


0.01 


OST program staffing 


34 


25 


3.26 


0.07 


State and local standards in 
reading, math, and/or science* 


34 


23 


4.88 


0.03 



* p<0.05 



Exhibit reads: Seventy percent of directors of schooi-based programs reported communicating with 
schooi staff at ieast once a month about the needs or progress of individuai students, compared to 52 
percent of directors of center-based programs. This difference was statisticaiiy significant. 
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Exhibit C5 

Staff Instructionai Strategies, by Activity Type 



Mean Score on Staff Instructional Strategies Scale (out of 7 points) 


Activity Type 


This Activity 


All Other Activities 


Mean Difference 


P 


Academic enrichment (n=59) 


3.62 


2.87 


0.74* 


0.00 


Arts (n=53) 


3.61 


2.90 


0.71* 


0.00 


Homework help/tutoring (n=81) 


2.68 


3.26 


-0.58* 


0.00 


Sports (n=20) 


2.46 


3.11 


-0.65* 


0.01 


Open, unstructured time (n=24) 


2.21 


3.16 


-0.95* 


0.00 



* p<0.05 



Exhibit reads: On average, academic enrichment activities were rated 3.62 out of 7 points on the Staff 
instructionai Strategies scaie, compared to an average rating of 2.87 for other activities observed. This 
difference of 0.74 points was statisticaiiy significant. 



Exhibit C6 

Quaiity of Activity Content and Structure, by Activity Type 



Mean Score on Activity Content and Structure Scale (out of 7 points) 


Activity Type 


This Activity 


Ail Other Activities 


Mean Difference 


P 


Arts (n=53) 


4.88 


4.00 


0.88* 


0.00 


Academic Enrichment (n=59) 


4.56 


4.11 


0.44 


0.10 


Sports (n=20) 


4.03 


4.29 


-0.27 


0.37 


Open, unstructured time (n=24) 


2.98 


4.46 


-1.49* 


0.00 



* p<0.05 



Homework activities are not inciuded in this exhibit because observers did not rate them on the content 
and structure items. 

Exhibit reads: On average, arts activities were rated 4.88 points out of 7 on the Activity Content and 
Structure Scaie, compared to an average activity rating of 4.00 for other activities observed. This 
difference of 0.88 points was statisticaiiy significant. 
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Exhibit C7 

Quaiity of Staff and Student Reiationships, by Activity Type 



Mean Score on Relationships Scale (out of 7 points) 


Activity Type 


This Activity 


All Other Activities 


Mean Difference 


P 


Sports (n=20) 


5.59 


5.28 


0.31 


0.17 


Arts (n=53) 


5.49 


5.26 


0.24 


0.11 


Academic Enrichment (n=59) 


5.44 


5.27 


0.17 


0.24 


Open, unstructured time (n=24) 


5.35 


5.31 


0.04 


0.83 


Homework Help/Tutoring (n=81) 


4.97 


5.49 


-0.52* 


0.00 



* p<0.05 



Exhibit reads: On average, sports activities were rated 5.59 points out of 7 on the Reiationships scaie, 
compared to an average activity rating of 5.28 for other activities observed. This difference of 0.31 points 
was not statisticaiiy significant. 
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Appendix D 

Details of Data Used in Analyses of Program Features 



Exhibit D1 

Participant Reports of Exposure to New Experiences, by Grade Levei 



Percent of Participants Who “Agreed a Lot” 


Bonferroni Test 


in this program... 


Eiementary-grades 

Participants 

(n=1,801) 


Middle-grades 

Participants 

(n=1,029) 


High School 
Participants 
(n=752) 


Comparison 


Sig. 


The activities really get me 
interested 


59 


56 


51 


Elementary-Middle 


0.34 


Middle-High 


0.10 


Elementary-High 


0.00* 


1 get a chance to do a lot of new 
things 


54 


53 


52 


Elementary-Middle 


1.00 


Middle-High 


1.00 


Elementary-High 


1.00 


There is a lot for me to choose to 
do 


48 


52 


42 


Elementary-Middle 


0.12 


Middle-High 


0.00* 


Elementary-High 


0.01* 


1 get to work on projects that really 
make me think 


46 


34 


36 


Elementary-Middle 


0.00* 


Middle-High 


1.00 


Elementary-High 


0.00* 


1 get to do things that 1 don't usually 
get to do anywhere else 


42 


39 


46 


Elementary-Middle 


0.71 


Middle-High 


0.03* 


Elementary-High 


0.20 



* p < .05 



Exhibit reads: Fifty-nine percent of eiementary-grades participants “agreed a iot” that their OST program activities reaiiy got 
them interested, compared to 56 percent of middie-grades participants and 51 percent of high schooi participants. 
Eiementary-grades participants differed significantiy from high schooi participants. 



Exhibit D2 

Participant Reports of Exposure to New Experiences, by Program Location 



Percent of Participants Who “Agreed a Lot” 


In this program... 


School-based 

Programs 

(n=2,721) 


Center-based 
Programs (n=860) 


Chi-Square 


P 


The activities really get me 
interested* 


59 


49 


23.54 


0.00 


1 get a chance to do a lot of new 
things 


53 


54 


0.11 


0.75 


There is a lot for me to choose to 
do* 


51 


40 


32.26 


0.00 


1 get to do things that 1 don’t usually 
get to do anywhere else 


42 


42 


0.02 


0.91 


1 get to work on projects that really 
make me think* 


40 


43 


4.02 


0.05 



* p < .05 



Exhibit reads: Fifty-nine percent of participants in schooi-based programs “agreed a iot” that their OST 
program activities reaiiy got them interested, compared to 49 percent of participants in center-based 
programs. The Chi-Square statistic was 23.54 with a p-value of 0.00. This difference was statistically 
significant. 



D-1 

































Exhibit D3 

Eiementary-grades Participant Reports of Interactions with Peers, 

by Program Location 



Percent of Participants Who “Agreed a Lot” 


in this program i... 


Schooi-based 

Programs 

(n=1,376) 


Center-based 
Programs (n=426) 


Chi-Square 


P 


Have a good time playing with other 
kids* 


71 


67 


3.16 


0.08 


Have a lot of friends 


70 


65 


3.80 


0.05 


Get to know other kids really well 


63 


67 


2.56 


0.12 


Get along with other kids 


51 


53 


0.34 


0.57 


Like the other kids 


40 


43 


1.33 


0.25 


Can really trust the other kids 


26 


29 


1.11 


0.32 



* p < .05 

Exhibit reads: Among eiementary-grades participants, 71 percent in schooi-based programs “agreed a iot” 
that they have a good time piaying with other kids in the program, compared to 67 percent of participants in 
center-based programs. This difference was statisticaily significant. 



Exhibit D4 

Participant Reports of Prosociai Behaviors, by Grade Levei 



Percent of Participants Who Reported Doing the Activity at Least Once 


in past 30 days i... 


High Schooi 
Participants 
(n=747) 


Middie-grades 

Participants 

(n=1,026) 


Chi-Square 


P 


Cooperated with others in 
completing a task 


87 


87 


0.70 


0.87 


Gave someone a compliment 


87 


86 


1.36 


0.72 


Helped other students solve a 
problem 


85 


85 


4.91 


0.18 


Told other students how 1 felt when 
they did something 1 liked* 


80 


71 


19.98 


0.00 


Told other students how 1 felt when 
they upset me* 


76 


67 


28.63 


0.00 


Helped someone stay out of a fight 


64 


62 


5.87 


0.12 


Protected someone from a bully* 


53 


59 


18.47 


0.00 



* p< .05 



Exhibit reads: Eighty-seven percent of high schooi participants reported that in the past 30 days they 
cooperated with others in compieting a task, compared to 87 percent of middie-grades participants. This 
reiationship was not statisticaily significant. 
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Exhibit D5 

Participant Reports of Sense of Beionging, by Program Location 



Percent of Participants Who “Agreed a Lot” 


in this program i feel like... 


School-based 

Programs 

(n=2,707) 


Center-based 
Programs (n=859) 


Chi-Square 


P 


I am safe* 


73 


78 


9.30 


0.00 


This is a good place to hang out 


60 


60 


0.08 


0.78 


I belong* 


59 


64 


7.61 


0.01 


I am successful 


56 


59 


2.49 


0.12 


I matter* 


55 


60 


7.64 


0.01 


My ideas count* 


45 


51 


10.40 


0.00 



* p< .05 



Exhibit reads: Seventy-three percent of participants in schooi-based programs “agreed a iot” that they feit safe 
in the program, compared to 78 percent of participants in center-based programs. This difference was 
statisticaliy different. 



Exhibit D6 

Participant Reports of Sense of Beionging, by Grade Levei 



Percent of Participants Who “Agreed a Lot” 


Bonferroni Test 


In this program 1 feel like... 


Elementary-grades 

participants 

(n=1,793) 


Middle-grades 

participants 

(n=1,026) 


High School 
participants 
(n=747) 


Comparison 


Sig. 


1 am safe 


76 


71 


74 


Elementary-Middle 


0.00* 


Middle-High 


0.29 


Elementary-High 


0.77 


1 belong 


60 


60 


60 


Elementary-Middle 


1.00 


Middle-High 


1.00 


Elementary-High 


1.00 


This is a good place to hang out 


59 


60 


61 


Elementary-Middle 


1.00 


Middle-High 


1.00 


Elementary-High 


0.78 


1 am successful 


57 


58 


57 


Elementary-Middle 


1.00 


Middle-High 


1.00 


Elementary-High 


1.00 


1 matter 


55 


56 


58 


Elementary-Middle 


1.00 


Middle-High 


0.80 


Elementary-High 




My ideas count 


46 


44 


50 


Elementary-Middle 


0.53 


Middle-High 


0.02* 


Elementary-High 


0.19 



* p< .05 



Exhibit reads: Seventy-six percent of eiementary-grades participants “agreed a iot” that they feit safe in the program, 
compared to 71 percent of middie-grades participants and 74 percent of high schooi participants. Eiementary-grades 
participants differed significantiy from middie-grades participants. 
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Exhibit D7 

Participant Reports of Interactions with Staff, by Program Location 



Percent of Participants Who “Agreed a Lot” 


in this program... 


Schooi-based 

Programs 

(n=2,712) 


Center-based 
Programs (n=861) 


Chi-Square 


P 


Staff treat me with respect* 


66 


73 


11.59 


0.00 


Staff think 1 can learn new things* 


66 


70 


4.34 


0.02 


Staff think 1 can do things well* 


62 


68 


10.22 


0.00 


Staff help me to try new things 


61 


62 


0.23 


0.66 


Staff always try to be fair 


60 


60 


0.02 


0.90 


Staff really care about me* 


58 


66 


15.82 


0.00 


1 feel that 1 can talk to staff about 
things that are bothering me 


51 


53 


1.06 


0.16 


Staff care what 1 think* 


49 


55 


8.42 


0.00 


Staff always keep their promises* 


44 


47 


3.66 


0.03 



* p< .05 

Exhibit reads: Sixty-six percent of participants in schooi-based programs “agreed a iot” that staff treated them 
with respect in the program, compared to 73 percent of participants in center-based programs. This 
difference was statisticaiiy significant. 
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Exhibit D8 

Participant Reports of Interactions with Staff, by Grade Levei 



Percent of Participants Who “Agreed a Lot” 


Bonferroni Test 


in this program... 


Eiementary-grades 

Participants 

(n=1,795) 


Middle-grades 

Participants 

(n=1,031) 


High School 
Participants 
(n=747) 


Comparison 


Sig. 


Staff treat me with respect 


68 


66 


70 


Elementary-Middle 


0.51 


Middle-High 


0.32 


Elementary-High 


1.00 


Staff think 1 can learn new things 


68 


63 


69 


Elementary-Middle 


0.01* 


Middle-High 


0.02* 


Elementary-High 


1.00 


Staff think 1 can do things well 


65 


60 


64 


Elementary-Middle 


0.03* 


Middle-High 


0.35 


Elementary-High 


1.00 


Staff always try to be fair 


64 


56 


57 


Elementary-Middle 


0.00* 


Middle-High 


1.00 


Elementary-High 


0.00* 


Staff help me to try new things 


64 


55 


63 


Elementary-Middle 


0.00* 


Middle-High 


0.00* 


Elementary-High 


1.00 


Staff really care about me 


62 


55 


61 


Elementary-Middle 


0.00* 


Middle-High 


0.02* 


Elementary-High 


1.00 


1 feel that 1 can talk to staff about 
things that are bothering me 


55 


47 


52 


Elementary-Middle 


0.00* 


Middle-High 


0.12 


Elementary-High 


0.27 


Staff care what 1 think 


50 


48 


55 


Elementary-Middle 


0.68 


Middle-High 


0.00* 


Elementary-High 


0.06 


Staff always keep their promises 


47 


38 


48 


Elementary-Middle 


0.00* 


Middle-High 


0.00* 


Elementary-High 


1.00 



* p< .05 



Exhibit reads: Sixty-eight percent of eiementary-grades participants “agreed a iot” that staff treat them with respect, compared 
to 66 percent of middie-grades participants and 70 percent of high schooi participants. There were no significant differences 
by grade ievei. 
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Exhibit D9 

High Schooi Participant Reports of Interactions with Staff, by Program Location 



Percent of Participants Who “Agreed a Lot” 


in this program... 


Schooi-based 
Programs (n=390) 


Center-based 
Programs (n=358) 


Chi-Square 


P 


Staff think I can learn new things 


68 


70 


0.26 


0.63 


Staff treat me with respect* 


65 


74 


6.45 


0.01 


Staff help me to try new things 


63 


65 


0.33 


0.59 


Staff think I can do things well* 


59 


68 


6.69 


0.01 


Staff really care about me* 


55 


68 


12.58 


0.00 


Staff always try to be fair 


54 


59 


2.02 


0.16 


Staff care what I think* 


50 


62 


10.95 


0.00 


I feel that I can talk to staff about 
things that are bothering me 


49 


54 


2.03 


0.16 


Staff always keep their promises* 


44 


52 


5.12 


0.03 



* p < .05 



Exhibit reads: Among high schooi participants, 68 percent of youth in schooi-based programs “agreed a iot” 
that staff thought they couid iearn new things, compared to 70 percent of youth in center-based programs. 
This difference was not statisticaily significant. 



Exhibit D10 

Participant Reports of Opportunities for Youth Leadership, by Grade Levei 



Percent of Participants Who Reported Doing the Activity 


At this program, i have... 


High Schooi 
Participants 
(n=741) 


Middie-grades 

Participants 

(n=1,027) 


Chi-Square 


P 


Led an activity* 


58 


51 


8.55 


0.00 


Helped out in the office* 


54 


26 


148.66 


0.00 


Helped out on a youth council, 
advisory group, or leadership team 
for this program* 


49 


30 


64.77 


0.00 


Helped plan a program activity or 
event* 


57 


37 


65.30 


0.00 


Helped with meetings for parents or 
community members* 


39 


20 


75.95 


0.00 


Been asked by staff for my ideas 
about the program or an activity* 


67 


56 


22.20 


0.00 



* p < .05 



Exhibit reads: Fifty-eight percent of high schooi participants reported ieading an activity in the program, 
compared to 51 percent of middie-grades participants. This difference was statisticaiiy significant. 
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Exhibit D11 

Participant Reports of Academic Benefits, by Grade Levei 



Percent of Participants Who “Agreed a Lot” 


Bonferroni Test 


This program has heiped me... 


Eiementary-grades 

Participants 

(n=1,801) 


Middle-grades 

Participants 

(n=1,032) 


High school 
Participants 
(n=747) 


Comparison 


Sig. 


Finish my homework more often 


67 


58 


33 


Elementary-Middle 


0.00* 


Middle-High 


0.00* 


Elementary-High 


0.00* 


Feel better about my schoolwork 


53 


35 


31 


Elementary-Middle 


0.00* 




0.17 


Elementary-High 


0.00* 


Get better grades in school 


52 


37 


32 


Elementary-Middle 


0.00* 


Middle-High 


0.11 


Elementary-High 


0.00* 


Solve math problems better 


52 


37 


25 


Elementary-Middle 


0.00* 




0.00* 


Elementary-High 


0.00* 


Read and understand better 


51 


34 


32 


Elementary-Middle 


0.00* 


Middle-High 


1.00 


Elementary-High 


0.00* 


Write better 


45 


33 


32 


Elementary-Middle 


0.00* 




1.00 


Elementary-High 


0.00* 


Use computers to do schoolwork 
better 


29 


30 


35 


Elementary-Middle 


1.00 


Middle-High 


0.13 


Elementary-High 


0.01* 



* p< .05 



Exhibit reads: Sixty-seven percent of eiementary-grades participants “agreed a iot” that the program has heiped them finish 
their homework more often, compared to 58 percent of middie-grades participants and 33 percent of high schooi participants. 
These differences were statisticaily significant between ali grade ieveis. 
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Exhibit D12 

Participant Reports of Academic Benefits, by Program Location 



Percent of Participants Who “Agreed a Lot” 


This program has helped 
me... 


Schooi-based 

Programs 

(n=2,721) 


Center-based 

Programs 

(n=859) 


Chi- 

Square 


P 


Finish my homework more 
often* 


60 


48 


39.76 


0.00 


Feel better about my 
schoolwork 


44 


42 


1.45 


0.12 


Get better grades in school 


44 


42 


1.92 


0.17 


Solve math problems better* 


44 


36 


15.69 


0.00 


Read and understand better 


43 


40 


1.61 


0.22 


Write better 


38 


40 


0.48 


0.49 


Use computers to do 
schoolwork better 


30 


33 


2.28 


0.13 



* p < .05 

Exhibit reads: Sixty percent of participants in schooi-based programs “agreed a iot” that their OST 
program has heiped them finish their homework more often, compared to 48 percent of participants 
center-based programs. This difference was statisticaily significant. 





Exhibit D13 

Participant Reports of Academic Seif-esteem, by Grade Levei 



Percent of Participants Who “Agreed a Lot” 


Bonferroni Test 


in generai i... 


Eiementary-grades 

participants 

(n=1,779) 


Middle-grades 

participants 

(n=1,018) 


High School 
participants 
(n=739) 


Comparison 


Sig. 


Try hard in school 








Elementary-Middle 


0.00* 


81 


68 


51 


Middle-High 


0.00* 










Elementary-High 


0.00* 


Do well in school 








Elementary-Middle 


0.00* 


66 


54 


39 


Middle-High 


0.00* 










Elementary-High 


0.00* 


Pay attention in class 








Elementary-Middle 


0.00* 


64 


49 


42 


Middle-High 


0.01* 










Elementary-High 


0.00* 


Enjoy math 








Elementary-Middle 


0.00* 


64 


45 


34 


Middle-High 


0.00* 










Elementary-High 


0.00* 










Elementary-Middle 


0.00* 


Always come to class prepared 


63 


49 


42 


Middle-High 


0.02* 










Elementary-High 


0.00* 










Elementary-Middle 


0.00* 


Always finish my homework 


63 


47 


33 


Middle-High 


0.00* 










Elementary-High 


0.00* 


Enjoy writing 








Elementary-Middle 


0.00* 


58 


42 


35 


Middle-High 


0.01* 










Elementary-High 


0.00* 










Elementary-Middle 


0.00* 


Enjoy school 


57 


38 


33 


Middle-High 


0.10 










Elementary-High 


0.00* 










Elementary-Middle 


0.00* 


Enjoy reading books for pleasure 


57 


32 


33 


Middle-High 


1.00 










Elementary-High 


0.00* 



* p< .05 



Exhibit reads: Eighty-one percent of eiementary-grades participants “agreed a iot” that they tried hard in schooi, compared to 
68 percent of middie-grades participants and 51 percent of high schooi participants. These differences were statisticaily 
significant between aii grade ieveis. 
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Exhibit D14 

Participant Reports of Academic Seif-esteem, by Program Location 



Percent of Participants Who “Agreed a Lot” 


in generai i... 


Schooi-based 

Programs 

(n=2,686) 


Center-based 

Programs 

(n=849) 


Chi- 

Square 


P 


Try hard in school* 


74 


63 


31.90 


0.00 


Do well in school* 


58 


52 


9.14 


0.00 


Pay attention in class* 


56 


51 


6.76 


0.01 


Enjoy math* 


54 


46 


16.56 


0.00 


Always come to class 
prepared* 


56 


49 


12.22 


0.00 


Always finish my hemewerk* 


54 


47 


11.71 


0.00 


Enjpy writing 


49 


48 


0.15 


0.72 


Enjoy school* 


48 


41 


14.14 


0.00 


Enjoy reading books for 
pleasure* 


46 


42 


4.27 


0.04 



* p < .05 



Exhibit reads: Seventy-four percent of participants in schooi-based programs “agreed a iot” that they 
tried hard in schooi, compared to 63 percent of participants in center-based programs. This difference 
was statisticaliy significant. 
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Exhibit D15 

High Schooi Participant Reports of Academic Seif-esteem, by Program Location 



Percent of Participants Who “Agreed a Lot” 


in generai i... 


Schooi-based 

Programs 

(n=388) 


Center-based 

Programs 

(n=351) 


Chi- 

Square 


P 


Try hard in school* 


57 


45 


10.99 


0.00 


Pay attention in class* 


45 


39 


3.45 


0.07 


Always come to class 
prepared 


44 


39 


2.23 


0.16 


Do well in school 


41 


36 


2.49 


0.13 


Enjoy writing 


37 


33 


1.41 


0.25 


Enjoy reading books for 
pleasure* 


37 


29 


5.36 


0.02 


Enjoy math 


36 


30 


2.95 


0.10 


Enjoy school* 


36 


29 


4.34 


0.04 


Always finish my homework 


35 


30 


1.66 


0.21 



* p < .05 



Exhibit reads: Among high schooi participants, 57 percent of youth in schooi-based programs “agreed a 
iot” that they tried hard in school, compared to 45 percent of youth in center-based programs. This 
difference was statistically significant. 



Exhibit D16 

Participant Reports of Skiii Deveiopment, by Grade Levei 



Percent of Participants Who “Agreed a Lot” 


Bonferroni Test 


This program has heiped me... 


Eiementary-grades 

Participants 

(n=1,772) 


Middle-grades 

Participants 

(n=1,009) 


High School 
Participants 
(n=735) 


Comparison 


Sig. 


Am good at using a computer 








Elementary-Middle 


0.04* 




64 


59 


48 


Middle-High 


0.00* 










Elementary-High 


0.00* 


Have the skills to be a leader 








Elementary-Middle 


0.09 




58 


54 


45 


Middle-High 


0.00* 










Elementary-High 


0.00* 


Can play a musical instrument, 








Elementary-Middle 


0.00* 


sing, dance, draw, paint, or do 


68 


59 


46 


Middle-High 


0.00* 


some other type of art really well 








Elementary-High 


0.00* 



* p< .05 



Exhibit reads: Sixty-four percent of elementary-grades participants “agreed a lot” that they were good at using a computer, 
compared to 59 percent of middle-grades participants and 48 percent of high school participants. These differences were 
statistically significant between all grade levels. 
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Exhibit D17 

Participant Reports of Skiii Deveiopment, by Program Location 



Percent of Participants Who “Agreed a Lot” 


in generai i... 


School-based 

Programs 

(n=2,676) 


Center-based 

Programs 

(n=842) 


Chi- 

Square 


P 


Am good at using a 
computer* 


61 


56 


5.71 


0.02 


Have the skills to be a leader 


55 


52 


3.31 


0.72 


Can play a musical 
instrument, sing, dance, 
draw, paint, or do some other 
type of art really well* 


62 


56 


12.95 


0.00 



* p< .05 



Exhibit reads: Sixty-one percent of participants in schooi-based programs “agreed a iot” that they were 
good at using a computer, compared to 56 percent of participants in center-based programs. This 
difference was statisticaiiy significant. 
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Appendix E 

Details of Associations Between 
Participant Experiences and Program Features 



Participant Survey Scaies 

Academic Benefits of the Program 

The Academic Benefits of the Program scale was computed to range from one to four, with four 
indicating that on average participants strongly agreed with the following statements: 

This program has helped me. . . 

■ Get better grades in school 

■ Feel better about my schoolwork 

■ Read and understand better 

■ Solve math problems better 

■ Finish my homework more often 

■ Write better 

■ Use computers to do schoolwork better 
Descriptive Statistics: 



Alpha 


Mean 


Standard 

Deviation 


Minimum 


25* 

Pereentile 


75* 

Pereentile 


Maximum 


0.84 


3.02 


0.72 


1 


2.67 


3.50 


4 




Disagree a lot 



Scale score 



Agree a lot 



E-1 



Academic Self-Esteem 

The Academic Self-Esteem scale was computed to range from one to four, with four indicating 
that on average participants strongly agreed with the following statements: 

In general I... 



■ Try hard in school 

■ Pay attention in class 

■ Always come to class prepared 

■ Enjoy school 

■ Enjoy reading books for pleasure 

■ Enjoy math 

■ Enjoy writing 

■ Always finish my homework 

■ Do well in school 

Descriptive Statistics: 



Alpha 


Mean 


Standard 

Deviation 


Minimum 


25* 

Pereentile 


75* 

Pereentile 


Maximum 


0.82 


3.32 


0.55 


1 


3.00 


3.78 


4 




1.0 1.5 2.0 2.5 3.0 3.5 4.0 



Disagree a lot Scale score Agree a lot 



E-2 



Interactions with Staff 

The Interactions with Staff scale was computed to range from one to four, with four indicating 
that on average participants strongly agreed with the following statements: 

In this program... 

■ Staff treat me with respect 

■ I feel that I can talk to staff about things that are bothering me 

■ Staff really care about me 

■ Staff always keep their promises 

■ Staff care what I think 

■ Staff always try to be fair 

■ Staff think I can do things well 

■ Staff help me to try new things 

■ Staff think I can learn new things 

Descriptive Statistics: 



Alpha 


Mean 


Standard 

Deviation 


Minimum 


25* 

Pereentile 


75* 

Pereentile 


Maximum 


0.91 


3.38 


0.65 


1 


3.00 


3.89 


4 




1.0 1.5 2.0 2.5 3.0 3.5 4.0 

Disagree a lot Scale score Agree a lot 
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Exposure to New Experiences 

The Exposure to New Experienees seale was computed to range from one to four, with four 
indicating that on average participants strongly agreed with the following statements: 

In this program. . . 

■ I get a chance to do a lot of new things 

■ I get to do things that I don’t usually get to do anywhere else 

■ I get to work on projects that really make me think 

■ There is a lot for me to choose to do 

■ The activities really get me interested 

Descriptive Statistics: 



Alpha 


Mean 


Standard 

Deviation 


Minimum 


25* 

Pereentile 


75* 

Pereentile 


Maximum 


0.71 


3.23 


0.61 


1 


3.00 


3.60 


4 




1.0 1.5 2.0 2.5 3.0 3.5 4.0 

Disagree a lot Scale score Agree a lot 
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Sense of Belonging 

The Sense of Belonging scale was computed to range from one to four, with four indicating that 
on average participants strongly agreed with the following statements: 

In this program I feel like. . . 

■ I belong 

■ My ideas count 

■ lam successful 

■ This is a good place to hang out 

■ I matter 

■ I am safe 

Descriptive Statistics: 



Alpha 


Mean 


Standard 

Deviation 


Minimum 


25* 

Pereentile 


75* 

Pereentile 


Maximum 


0.81 


3.41 


0.59 


1 


3.00 


3.83 


4 




1.0 1.5 2.0 2.5 3.0 3.5 4.0 



Disagree a lot Scale score Agree a lot 
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Interactions with Peers 



The Interactions with Peers scale was computed to range from one to four, with four indicating 
that on average elementary-grades participants strongly agreed with the following statements: 

In this program I... 

■ Get to know other kids really well 

■ Can really trust the other kids 

■ Have a lot of friends 

■ Like the other kids 

■ Have a good time playing with other kids 

■ Get along with other kids 

Descriptive Statistics: 



Alpha 


Mean 


Standard 

Deviation 


Minimum 


25* 

Pereentile 


75* 

Pereentile 


Maximum 


0.79 


3.30 


0.60 


1 


3.00 


3.83 


4 



w 

( 1 ) 

■o 

m 



c 

0) 



0) 

( 1 ) 



o 



c 

( 1 ) 

u 

( 1 ) 
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1.0 



Disagree a lot 



1.5 2.0 2.5 

Scale score 



3.0 3.5 4.0 

Agree a lot 
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Exhibit E1 

Correiations between Program Focus on 
Academic Activities in Programs and Participant Experiences 



Participant Survey Scale 


Pearson 

Correlation 


P 


n 


Academic Benefits* 


0.37 


0.00 


89 


Academic Self-esteem* 


0.25 


0.02 


89 


Interactions with Peers' 


0.10 


0.52 


44 


Exposure to New 
Experiences 


-0.07 


0.52 


89 


Sense of Belonging 


-0.16 


0.13 


89 


Interactions with Staff* 


-0.23 


0.03 


89 



* p< .05 

^This scale reflects elementary-grades participants only. 

Exhibit reads: The extent to which youth reported academic benefits of the program was positively 
associated with the Intensity of academic activities in a program. The Pearson Correlation was 
0.37 with a p-value of 0.00. This correlation was statistically significant. 



Exhibit E2 

Correiations between Program Focus on 
Arts Activities in Programs and Participant Experiences 



Participant Survey Scale 


Pearson 

Correlation 


P 


n 


Academic Benefits* 


0.23 


0.03 


89 


Academic Self-esteem* 


0.23 


0.03 


89 


Interactions with Peers' 


0.20 


0.20 


44 


Exposure to New 
Experiences 


0.10 


0.37 


89 


Sense of Belonging 


-0.01 


0.92 


89 


Interactions with Staff 


-0.10 


0.34 


89 



* p< .05 

^This scale reflects elementary-grades participants only. 



Exhibit reads: The extent to which youth reported academic benefits of the program was positively 
associated with the Intensity of arts activities in the program. The Pearson Correlation was 0.23 
with a p-value of 0.03. This difference was statistically significant. 
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Exhibit E3 

Correiations between Program Focus on 
Physicai Activities in Programs and Participant Experiences 



Participant Survey Scale 


Pearson 

Correlation 


P 


n 


Interactions with Peers' 


0.06 


0.71 


44 


Academic Benefits 


-0.02 


0.84 


89 


Academic Self-esteem 


-0.05 


0.63 


89 


Sense of Belonging 


-0.19 


0.07 


89 


Exposure to New 
Experiences* 


-0.25 


0.02 


89 


Interactions with Staff* 


-0.32 


0.00 


89 



* p< .05 

^This scale reflects elementary-grades participants only. 



Exhibit reads: The extent to which elementary-grades participants reported having positive 
interactions with their peers was positively associated with the intensity of physical activities In the 
program. The Pearson Correlation was 0.06 with a p-value of 0.71 . This correlation was not 
statistically significant. 



Exhibit E4 

Correiations between Participation Rates and Participant Experiences 



Participant Survey Scale 


Pearson 

Correlation 


P 


n 


Academic Self-esteem* 


0.38 


0.00 


87 


Academic Benefits 


0.17 


0.13 


87 


Interactions with Peers' 


0.09 


0.58 


43 


Exposure to New 
Experiences 


-0.02 


0.85 


87 


Interactions with Staff 


-0.08 


0.48 


87 


Sense of Belonging 


-0.10 


0.37 


87 



* p<05 

^Thls scale reflects elementary-grades participants only. 

Exhibit reads: The extent to which participants reported positive academic self-esteem was 
positively associated with higher participation rates in the program. The Pearson Correlation was 
0.38 with a p-value of 0.00. This correlation was statistically significant. 
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